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ABSTRACT 

The P.E.P. Report 1969-1973 focuses on the various 
findings and activities of the Program Evaluation Project. Follovup 
is an integral part of a goal oriented evaluation, providing the 
opportunity for the collection of various forms of outcome data as 
veil as consumer satisfaction information. This chapter discusses the 
history and implementation of the follovup program for the Program 
Evaluation Project, including the development of a questionnaire, 
training of the intervievers, locating and contacting participants, 
and costs. The second section discusses the various studies in vhich 
the follovup unit vas involved, including the original reliability 
study in vhich clients were followed up by two different interviewers 
at two distinct points in time, the interdisciplinary reliability 
study which discusses the comparison of R.N. and M.S.i. interviewers 
and phone versus in-person interviews, the followup program for the 
Crisis Intervention Center, and the followup of clients terminated 
from the Hennepin County Mental Service prior to involvement in 
treatment. (Author/RC) 
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GENERAL INTRODUCTION TO THE P.E.P. REPORT 1969-1973 



The P.E>P. Report 1969-1973 focuses on the various findings and activities of the Program Evaluation 
Project. It is being published in pamphlet fom with one pamphlet for each chapter. 

As of January, 1974, the Program Evaluation Project, whose title was changed to the Program Evaluation 
Resource Center as of June, 1974, is funded by a three year collaborative grant with the Mental Health 
Services Division of the National Institute of Mental Health. The purpose of the grant is to emphasize the 
coordination and dissemination of information on a variety of program evaluation methodologies, especially 
Goal Attainment Scaling. 

Further information on the Goal Attainment Scaling methodology and program evaluation is available in 
other written and recorded materials from the Program Evaluation Resource Center office. At this time 
various other chapters of the P.E,P. Report 1969-1973 are available, including Chapter One, "Basic Goal At- 
tainment Scaling Procedures", Chapter Three, "An Introduction to Reliability and the Goal Attainment Scaling 
Methodology", Chapter Four, "An Examination of the Reliability of the Kiresuk-Sherman Goal Attainment Score 
by Means of Components of Variance", Chapter Five, "A Construct Validity Overview of Goal Attainment Scaling" 
and Chapter Nine, "Evaluation of the Adult Outpatient Program, Hennepin County Mental Health Service". 
Additional chapters will be released this year as they are completed. 
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SYNOPSIS FOR CHAPTER TWO 
THE FOLLOW-UP UNIT 



PURPOSE- Follow-up is an integral part of a goal oriented evaluation, providing the opportunity for the 
collection of various forms of outcome data as well as consumer satisfaction information. This chapter- 
discusses the history and implementation of the follow-up program for the Program Evaluation Project, in- 
cluding the developtrent of a questionnaire, training of the .ntervi ewers, f"^ contac "9 Par- 
ticioants. and costs. The second section discusses the various studies in which the fol low-up urn t was 
involS including the original reliability study in which clients were followed up by two different 
Interviewers at two distinct points in time, the interdisciplinary reliability ^^u y which discusses t^^ 
comparison of R.N. and M.S.W. interviewers and phone versus in-person interviews, the follow-up program 
for the Crisis Intervention Center, and the follow-up of clients terminated from the Hennepir County Men- 
tal Health Service prior to involvement in treatment. 



SUMMARY: The follow-up questionnaire which was used consistently included both cons^ 



question s and questions relating to clients' level of attainment as reflected by the Goal Attainment 
Fonow-uo Guide Three revisions of the questionnaire have been produced. The follow-up interviewers 
were includedin the discussion of questionnaire revisions, since their experience conducting interviews 
was considered to be important in improving the format of the follow-up questionnaire. 

The original team of interviewers hired for the follow-up program were Master's level social workers 
After comp itinq more than a year of follow-up interviews, experimentetion with interviewers of d fferent 
SSucational and^p^tessio^al backgrounds was underteken. The follow-up unit employed B.A. leve inter- 
viewers, undergraduate university students majoring in the social sciences, registered nurses and 
teacher s aides from the Minneapolis school system. Of these, only the teacher's aides were not successful 
interviewers, and all were salaried at a rate less than the Master's level socal workers. 

During the four years of the Program Evaluation Project, 1128 follow-up interviewers were completed at 
a total cost ot $15,791 .98. The average cost per interview, for interviewer s salaries was $ 3.99, and 
?hemlan number of nterviews completed per year was 282. Included in these costs are not only payment to 
intervtewers L Llual completion of the 1128 interviews, but also costs resulting from training and ad- 
ministrative meetings with the follow-up interviewing staff as well. 

The interdisciplinary study compared in-person and phone interviews to determine if telephone inter- 
views which were less expensive to complete, would result in significantly difterent Goal Attainment 
scores The mean Goal Attainemnt score for the 52 interviews conducted in-person was 50.22 and the mean 
GoarAitainment score for the 68 interviews conducted by telephone was 52.90. No f ^3^^^" 

Goa Attainemnt scores reached the p > .05 level of statistical significance. Each of the cl ents par- 
ScJpated in two interviews conducted by a random combination of either R.N. or M.S.W. interviewers. In 
comparisons of interviews conducted by the nurses and those conducted by the social workers, no level ot 
significance equal to or greater than p > .05 was determined in terms of the mean Goal Attainment score. 

The original reliability study was designed to compare the reliability of the Goal Attainment scores 
when two protessionals independently prepared follow-up guides for the same client and also che reli- 
abilitv of two follow-up interviewers conducting independent interviews with the client. Two follow-up 
; i; 'were constJJcted'for each of the 44 clients involved in the study, one by the "ta e interviewer 
after two intake interviews, and the second by the therapist after completion of two therapy sessions. 
At the t?ml 0 follow-up, the scales from both follow-up guides were random y mixed 0 J''^', "^^ 

the clients were each interviewed by two difterent follow-up interviewers at two different sessions The 
follow-up guides were separated after the follow-up interviews for the purposes of the analysis. 
mean Goal Atteinment scores for the first follow-up of the inteke interviewer-constructed follow-up guide 
^s 48 62 with a standard deviation of 9.18 and for the therapist follow-.p gu de, 51.45 with a standard 
deviation of 9 84 The mean Goal Atteinment scores derived from the second follow-up interview were 49.83 
f^r the intake interviewers' follow-up guides, and 53.57 for the therapist follow-up guides. An analysis 
S varfaS was empSed ^0 determine that 18^. of the Goal Attainment score variance was due to follow-up 
interv ewer errors in scoring or observation, 17% due to the choice of follow-up guide materia , 15. due 
lo3hort-te™ client changes or follow-up bias flucteations, and 50% due to short-term client long-term 
deviation from expectation. 



Other studies in which the follcw-up unit was involved are also discussed. 
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I . History, Planning and Implementation of the 
Follow-Up Unit 



A. Objectives of the Follow-Up Unit 

As stated in the original grant application 
for the Program Evaluation Project, the purpose 
of follow-up for clients who received treatment 
from the Hennepin County Mental Health Service 
during the coursr of program evaluation there was 
"to establish continuous feedback on treatment ef- 
forts of the Mental Health Center." Specifically, 
its purpose was "to report on progress made with 
regard to those goals selected during the intake 
for each client and to report any new problems 
requiring attention that may have developed dur- 
ing... the period following treatment assignment." 
The authors of the Grant Proposal expected that 
the follow-up unit would provide the inormation 
base for the evaluation study and also benefit 
Individuals within the community who needed help. 

The original research design included therapy 
mode outcome comparison among Individual, Group, 
Chemotherapy, ai^d Day Treatment. Follow-up emphasis 
was placed primarily on clients who were assigned 
these treatment modes randomly, and only secondarily 
on as many clients assigned non-randomly to therapy 
as was monetarily possible. (See chapter or proce- 
dures of the four mode study in the P.E.P. Report , 
1969-1973. ) 

From the outset of the project, follow-up plans 
also included studies to determine: 1) the reli- 
ability of the Goal Attainment score when the cli- 
ent was followed up by two different interviewers, 
and 2) the reliability of the guide construction 
when the goals were established for the same cli- 
ent by two different guide constructors. (See 
Section IIA of this report.) 



B . Planning and Experimental Follow-Ups 

Follow-up unit staff began planning operations 
in January, 1970. Formal data collection began at 
the Mental Health Service on February 2, 1970. 
The first year follow-ups were scheduled for June of 
that year. 

The principal investigator of the project in- 
vited the chief psychiatric social worker from 
the Mental Health Service to coordinate the follow- 
up unit. They decided to employ as follow-up in- 
terviewers master's level social workers who would 
work on a moonlighting basis. The follow-up co- 
ordinator took responsibility for hiring, training, 
and supervising the original interviewing staff. 
The coordinator was also instrumental in the prep- 
aration of a uniform interview schedule. 

To determine what would be the most useful 
procedures for scoring the follow-up guides, re- 
search staff assembled an experimental follow-up 
interview questionnaire and scheduled experiment- 
al follow-ups for June of 1970 with clients either 
who were counseled out of therapy after their in- 
take interviews or who w^.re assigned to treatment 



but never returned after assignment. These clients, 
because they received no "therapy" as such from the 
Mental Health Service, were considered ineligible 
for inclusion in the four mode study. It was as- 
sumed, however, that these clients had sufficient 
contact with the Service to provide the information 
necessary for pre-testing the follow-up interview 
questionnaire. 

Fifteen clients were selected for participation 
in the experimental follow-up, and of these, 11 
were interviewed. The interviewers were Program 
Evaluation Project staff and Mental Health Service 
staff who were interested in planning for follow-up. 

Based on the experimental follow-ups, the par- 
ticipating interviewers revised those parts of the 
questionnaire related to consumer satisfaction and 
prepared instructions and clarifications of each 
question for the follow-up interviewers. (See 
Appendix, page 24.) 



C. The Interview Schedule 

The interview schedule contained two sections. 
The first section of the intervii// schedule, by 
which the follow-up interviewer determined the 
client's level of satisfaction with services from 
the Mental Health Service, was originally design- 
ed as an introduction and a "get acquainted time" 
for the interviewer and the client. However, once 
the research staff began looking at the consumer 
satisfaction data collection, they decided that 
useful comparisons could be derived from this 
information. The consumer satisfaction questions 
that were deemed most relevant for general attitud- 
inal comparisons among all clients followed ud 
were questions 4 through 8, 9, 11, and 11a. (See 
Appendix, pages 27-29 and 34.) These are the re- 
sponses that were consistently coded for data 
analysis from all interviews completed. 

The second half of the questionnaire, by hich 
the follow-up interviewer scored each scale the 
Goal Attainment Follow-up Guide at the appropriate 
level, focused on problems that brought the client 
to the Mental Health Service, and how he/she felt 
about those problems at the time of the follow-up 
interview. Because clients did not know a Goal 
Attainment follow-up Guide had been constructed for 
them (however, cl ients did sign a consent form 
giving permission for the research and follow-up 
activities in general), the interviewers were asked 
to elicit tne information necessary to score the 
follow-up guide without revealing the nature of 
the follow-up guide to the client. The semi- 
structured questionnaire seemed appropriate to this 
purpose and to establishing a degree of conformity 
and control over all interviews completed. 

The interview schedule, developed in June, 1970 
after the experimental follow-ups, was rc'^sed in 
November, 1970, after thefirst six months of fol- 
low-up. (See Appendix, page 26.) This second re- 
vision provided more space for listing responses 
to "consumer satisfaction" questions and requested 
that interviewers note in greater detail comments 
made by the client during the interview. The fol- 
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low-up interviewers themselves asked that the lay- 
out of the schedule be modified, as some questions 
could not b3 answered with a "yes" or "no'* response. 

The November, 1970 revision of the Interview 
format (see Appendix, page 32) was used until 
August, 1971, when a new revision was effected. 
In early 1972, page 6a (see Appendix, page 38) 
was added to the last revision to provide more 
specific Information about ways In which the client 
felt the Service could Improve Its functioning. 
At the same time the last page of this packet (see 
Appendix, page 38) was added as a formal way to 
Inform appropriate Mental Health Service staff of 
pfoblem situations manifested during the follow-up 
Interview that the Interviewer felt warranted fur- 
ther Interaction between the client and his thera- 
pist. 

Included In the Interview packet after Septem- 
ber, 1972, was the Goal Attainment Follow-up Guide 
Critique Form (See Figure I). On this critique 
form a therapist evaluated the "realism," "rele- 
vancy" and completeness of the Goal Attainment 
Follow-up Guide constructed for his or her client 
(usually by another clinician, the Intake Inter- 
viewer). At follow-up, the follow-up Interview- 
ers were also asked to assess the follow-up guides. 

bCLl: therapist and follow-up Interviewer had 
the option Indicating what. If any, new problem 
areas they would have added to the follow-up guide. 
The recommendations about the problems to be add- 
ed depended on the perspective of the person In- 
volved. The therapist would base judgments on new 
problem areas on their Interaction with the client 
during treatment, while the follow-up Interviewer 
would base his on the follow-up Interview 



The follow-up guide critique forms were filled 
out by the therapist approximately one month after 
the client was assigned to treatment and by the fol 
low-up interviewer after the follow-up interview. 
Data is available from both the therapist and the 
follow-up interviewer on the same clients for only 
a small sample (N=26). However, data was collected 
on 95 follow-up guides critiqued by the therapists 
from May, 1972 through December, 19/2. Another 105 
critique forms were completed by the follow-up in- 
terviewers after the follow-up interview, with a 
total of 338 scales assessed. 

The comparative analysis of these two groups 
shows very little agreement overall, between 
the therapist and follow-up interviewer assess- 
ments. T'-ie therapists left more questions un- 
answered than did the follow-up interviewers, 
which may account for a good deal of the dis- 
crepancy. 

Contained in the 95 follow-up guides assessed 
by the therapists were 334 scales. Of these, 272 
(81.44%) were deemed relevant in the therapists' 
opinion?, 178 (53.29%) were deemed realistic, and 
for 48 of the 95 follow-up guides, (50.53%), ad- 
ditional scale headings were suggested. 

The follow-up interviewers assessed 105 follow- 
up guides at the time of the follow-up interview, 
with a total of 338 scales. Of these. 315 (93.20%) 
we/e deemed relevant, 232 (68.64%) were seen as 
realistic, and for 42 of the 105 follow-up guides 
(4055), additional scale headings were suggested. 

Of the 26 clients for w^ich the follow-up guide 
critique form was completed by both the therapist 
and the follow-up interviewer, 21 follow-up guides 
were scored by both assessors. The five follow-up 



FIGURE I: Goal Attainment Follow-up Guide Critique Form 

Patient Name: Negot. 

Follow-up Interviewer: Guide No. 

Scale 1 Scale 2 Scale 3 Sca^e 4 Scale-5 

1) Are the scales relevant? Yes No Yes No Yes No Yes No Yes No 

2) Ar*^ the scale levels Optimistic Optimistic Optimistic Optimi«^tic Optimistic 
realistic? Right On Right On Right On Right On Right On 
(Circle One) Pessimistic Pessimistic Pessimistic Pessimistic Pessimistic 

3) Would you have added any scales to this guide? Yes No 

4) If you answered "yes*' to question 3, please indicate what additional scale headings 
or content areas of existing scales you could have added. 
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guides that were not scored were left blank by the 
tLerapist but were scored by the follow-up inter- 
viewer. The mean Goal Attainment score (excluding 
the Five clients' follow-up guides not scored by the 
therapists) for the follow-up guides scored by 
the follow-up interviewers is 55.58, and the mean 
Goal Attainment score for those follow-up guides 
scored by the therapists is 57.61. The correla- 
tion of the Goal Attainment scores from thera- 
pists and from follow-up interviewers for the 
21 follow-up guides is .57, as computed by the 
Pearson Product Moment method. 



All interviews were scheduled by the Program 
Evaluation Project staff and it was assumed that 
the average interview would not take longer than 
an hour to complete. Payment for comp< ^ted in- 
terviews was established at $15. For those in- 
terviews scheduled, but which the client failed to 
attend, the interviewers originally were paid $5; 
after three months of experience, however, the pay- 
ment for "failed" interviews was increased to $7.50 
to cover the interviewer's time preparing for and 
traveling to an interview. 



Table 1 reports the consistency of responses 
from the therapist critique forms and tho -f? com- 
pleted by the follow-up interviewers for u,ie 96 
scales which both assessed. In summary, the 
critique forms completed by both the therapists 
and ^ol low-up interviewers showed that in 80% 
or more, the scales were relevant, 53% or more 
were deemed realistic, and for 45% Of the guides, 
additional scale headings were suggested at the 
time the Guide Critique form was completed. 



TABLE I: 



N = 96 



Comparison of Threapist and Follow-up 
Interviewer Ratings on the Goal Attain- 
ment Follow-up Critique Form 



TmERAPIST 

optimistic Right On Pt'ss imi s t ic „o 



« OptfmlUlc 
5 Right On 
^ Pessimfstic 
Ho Answer 



2 
13 
1 
3 



1 

40 
4 

0 



1 

19 
1 
1 



D. The Fol low-Up Interviewers 

The follow-up interviewer was the key source 
of data collection for the research project. As 
stated, the original team of follow-up interview- 
ers were all M.S.W. social workers who were not 
employed by the Hennepin County General Hospital 
or, until that time, by the Program Evaluation 
Project. They were selected on the basis of their 
reputations as experienced and knowledgeable in- 
terviewers. It was assumed that having inter- 
viewers who were not affiliated with the Mental 
Health Service 1) would encourage less bias in 
the scoring of the Goal Attainment Follow-up. 
Guides, and 2) would allow the interviewer to tell 
the client that he wasn't part of the Mental Health 
Service staff and had no veFted interest in the re- 
sults of the follow-up interview, thus encouraging 
clients to be freer in their discussion of satis- 
faction/dissatisfaction with mental health services. 



Seven M.S.W.'s were hired as interviewers; 
one resigned due to other commitments prior to 
completing any follow-up interviews. Five of 
the remaining six interviewers were regularly 
employed by Hennepin County Court Services, 
Catholic Welfare, Family and Children's Service, 
and th^ Minneapolis School Board. The other 
interviewer was not employed in the community 
at the time. 



1. Orientation of the Interviewing Staff 

Near the end of June, 1970, the first meeting 
was held with the six follow-up interviewers to 
relate the results of the experimental follow-up 
and to familiarize the interviewers with the Pro- 
gram Evaluation Project objectives. The inter- 
view schedule was reviewed carefully and explain- 
ed in detail to the interviewers. The issue of 
confidentiality was stressed. The interviewers 
were reminded that the objective of a follow-up 
interview was to score the appropriate level for 
each problem area mentioned on the follow-up 
guide. The follow-up interviewers were to re- 
main unaware of a client's therapist and therapy 
mode unless the client revealed that information 
during the course of the interview. However, 
interviewers were told that the follow-up co- 
ordinator would later help them with any problems 
or questions they mignt have w^th scoring a fol- 
low-up guide. 

At the suggestion of follow-up interviewers, 
the contents of the follow-up guides were review- 
ed prior to an interview to avoid placing the 
female interviewers in a potentially dr^ngerous 
situation. It was decided that if the client's 
Goal Attainment Follow-up Guide indi'-^ced prob- 
lems such as rape, exhibitionism, violence, etc., 
every attempt would be made to assign the client 
to a male interviewer. 



2. Fol low-Up Interviewers Other than M.S.W.'s 

By January, 1971, the number of clients eligi- 
ble for follow-up had increased significantly, and 
a B.A. social worker was then added to the inter- 
viewing staff. This interviewer was not employed 
outsiae her home, and was available for daytime 
interviews, thus filling a need which had become 
apparent to the research staff. 

Beginning in midsurmier of 1971, Project staff 
sought to employ interviewers of various education- 
al and professional backgrounds to determine what 
kinds of persons besides M.S.W.'s could be used as 
interviewers. These interviewers were to be em- 
ployed at lower fees than the M.S.W.'s, but would 
still have to be effective interviewers and able 
to complete the scoring of the Goal Attainment 
Follow-up Guide. As one method of hiring interview- 
ers, the follow-up supervisor obtained a list of 
recently graduated bachelor-degreed nurses from 
the University of Minnesota and contacted them, ex- 
plaining the follow-up study and inquiring as to 
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their wmingntss to participate as follow-up in- 
terviewers. Ar: orientation session was held with 
the nurses to familiarize them with Goal Attain- 
ment Scaling and the interview schedule. At least 
one of the experiefnced M.S.W. follow-up inter- 
viewers attended orientation sessions for the new 
follow-up interviewers whenever they were held. 
The nurses had, because of their professional train 
ing, some experience as interviewers. Payment was 
established at $5 per completed interview for the 
first ten completed, and $7.50 per completed in- 
terview after the initial ten. The five nurses 
hired were used in an interdisciplinary follow-up 
study in which each client was interviewed twice, 
each time by a randomly assigned R.N. or M.S.W. ; 
these interviews were conducted (again, by ran- 
dom assignment) either in person or by phone. Re- 
sults of this study are discussed in Section II of 
this chapter. 

Another group of interviewers hired to parti- 
cipate in the same discipline comparison study 
were paraprofessional teacher's aides employed 
by the Minneapolis School System. The follow-up 
supervisor contacted several teacher's aides, 
giving much the same information as when the n^^rses 
were contacted. The teacher's aides were given 
the same orientation and instructional sessions, 
and were to be paid the same amount for completed 
interviews as the nurses. Although these orienta- 
tion sessions provided adequate information to the 
nurses regarding the research study design and 
purpose, it is speculated that these sessions were 
not sufficiently explicit for the paraprofession- 
als. Of the four teacher's aides who were origin- 
ally recruited, only two completed interviews and 
voth were unable to score the follow-up guide with 
confidence. Because of their lack of understand- 
ing of the purpose of the research design and in 
some cases di5iatisfaction with the rate of pay, 
all four resigned of their own accord. 

The nurses, who presumably had a better under- 
standing of the purposes of the study and more 
adequate interviewing skills, were also unable 
to continue with the project for very long, pri- 
marily because many of them were employed by area 
hospitals and worked rotating shifts, and they 
lacked the time for interviewing. From May through 
August, 1972, only two of the original five 
nurses were available for interviews, and by Au- 
gust, all had resigned due to lack of time. It 
should be noted, however, that both the nurses and 
the teacher's aides, unlike the original follow- 
up team, were requested to make their own appoint- 
ments for follow-up interviews. 

Undergraduate students from the University of 
Minnesota were also employed as interviewers at 
the same payment rate offered to the teacher's 
aides and the nurses. The students' recruitment, 
however, unlike that of either the teacher's 
aides or the nurses, was by an advertisement 
for follow-up interviewers placed in the Univer- 
sity of Minnesota's student newspaper. In light 
of the Project's experience with the teacher's 
aides, and the students' lack of past experience 
with interviewing, students' orientation was more 
intensive than that of previous groups. The or- 
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ientation involved explicit discussion of the In- 
terview schedule and the procedures for arranging 
interviews, and also included roie playing, with 
the help of one of the experienced M.S.W. follow- 
up interviewers. The students proved to t»e 
thorough interviewers, although two of the orig- 
inal four who were hired had to resign because 
of other time commitments. The students were 
originally hired to follow up clients from the 
Crisis Intervention Center of Hennepin County Gen- 
eral Hospital (See chapter on the Crisis Inter- 
vention Center in the P. £»P. Report. 1969-1973 ). 
Eventually they completed follow-ups for the four 
mode study as well. The students, like the teach- 
er's aides and the nurse5i were asked to arrange 
their own appointments for follow-up. 

It is interesting to note that the interview- 
ers who were asked to arrange their own appoint- 
ments for follow-up (nurses and students) had as 
much difficulty locating the clients as the Program 
Evaluation Project follow-up staff, but because 
they were requested to do so from the start, it 
proved to be no problem. On the other hand, when 
the original team of follow-up interviewers (M.S.W.) 
were asked to arrange their own appointments for 
the interdisciplinary study, though they complied 
with this requirement, they found it to be an un- 
welcome burden. 

The interdisciplinary reliability study com- 
pared not only different interviewer disciplines, 
but also compared in-person interviews with tele- 
phone interviews. Many of the experienced inter- 
viewers who had originally conducted only in-per- 
son interviews, found phone interviewing more 
difficult in terms of being able to score the 
Goal Attainment Follow-up Guide. Also, many of 
the experienced interviewers felt that a great 
deal was lost during the interview by not having 
a personal, face to face encounter with the cli- 
ent. The results of the study, however, showed 
little difference between interviews conducted in 
person and those done by phone, as reflected in 
the mean Goal Attainment scores, jhis study 1s 
discussed in detail in Section IlB of this 
report. 

3. Communications with Follow-up Interviewers 

Throughout the Project's involvement with fol- 
low-up studies, when a new substudy was to begin 
that would in any way alter the usual requirements 
for follow-up, a meeting of the interviewers with 
the research staff was held to explain the new 
study and discuss any changes that it required in 
the interview schedule. 

This method of communication with the follow-up 
interviewing staff proved to be very effective. An 
in-person meeting with all the interviewers also 
promoted consistency in interviewing methods and an 
exchange of ideas which would have been lost if 
communication was limited to individual contacts 
with interviewers via telephone or letter. The in- 
terviewers were paid for attendance at the follow- 
up staff meetings according to their usual rate of 
payment for completed follow-up interviews. 



The only major issue with which all the inter- 
viewers were concerned was the element of confi- 
dentiality that resulted from the initiation of 
formal feedback to the Mental Health Service cli- 
nicians, in February of 1972. During the early 
phases of follow-up, the interviewers were noti- 
fied that they could assume confidentiality of the 
follow-up interview, so that anything the client 
related regarding his therapist or therany, would 
not in turn be related to his therapist. The in- 
terviewers were adament in honoring that promise 
to the client, and as ? result, feedback data of 
these early follow-up interviews was reported to 
the Mental Health Service clinicians only in ag- 
gregate form, not by individual clients. From Jan- 
uary, 1972 on, however, the interviewers no longer 
promised that the client's therapist would not be 
made aware of the results of the interview. This 
change seemingly had no effect on the clients' 
willingness to discuss their level of satisfaction/ 
dissatisfaction regarding the services at the Men- 
tal Health Service. The mean consumer satisfaction 
index score for the 426 clients followed up prior 
to January, 1972 was 76.18, as compared tc the 
mean consumer satisfaction index 3^-0re of the 265 
clients followed up after January, 19/2, which was 
77.50. Since the clients seemed satisfied with 
treatment when the therapists were made aware of the 
client's results, it was assumed that the clients 
would accept therapist involvement in the follow-up 
process. As a result, the Mental Health Service 
redesign evaluation (see chapter on the redesign, 
P.E.P. Report, 1969-1973) primarily involves the 
therapist as follow-up interviewer to insure 
irrmediate clinical feedback. 



E. Implementation of Fol low-Up 

The criteria determining a client's eligibility 
for follow-up in the .our mode study included: 1) 
a Goal Attdinment Follow-up Guide completed by 
the intake interviewer after not more than two in- 
take sessions, 2) the client's attendance at a 
minimum of two therapy sessions with his assigned 
therapist, and 3) a signed consent form. Accurate 
record- keeping of clients visits to the Mental 
Health Service was imperative to determine whether 
or not these three criteria were met. 



1 . Tickler File 

To help determine when clients were potentially 
eligible for follow-up, follow-up staff establish- 
ed a tickler file listing all clients due for fol- 
low-up during a given month. The cards for the file 
were completed by a member of the research staff 
immediately after a client's assignment to therapy, 
but the cards were not placed in the file unti I a 
follow-up guide had been completed. In the case of 
a late follow-up guide being completed and routed 
to'the research office, the tickler card was com- 
pleted with the prescribed date of follow-up indi- 
cated and included in the follow-up tickler file. 



To determine a client's eligibility for follow- 
up it waj necessary to obtain course of theVapy in- 
formation from intake through the prescribed follow- 
up date. All clients for a given month were listed 
by a research staff member, who reviewed their Men- 
tal Health Service charts to determine intake in- 
terviewer, therapist, therapy mode, and number of 
times seen for treatment. This chart check served 
as a means of verifying the assignment as ^scordecJ 
by the research staff at the time of assignment, 
and also recognition of any transfers or termin- 
ations that were not reviewed by the Intake Coai- 
mittee. In the case of charts not available for 
review, a memo (see Appendix, page 45) was 
sent to the client's therapist asking him to in- 
dicate the number of times he'd seen the client, 
and how many more times he expected to meet with 
the client, if the client was still actively in- 
volved in treatment. 

The original method of determining how many 
times and for what services a client was seen 
at the Mental Health Service was to collect daily 
copies of the Outpatient Department Transaction 
Record, filled out by the receptionists at the 
clinic desk. After February, 1970, however, the 
research staff recorded the new clients' visits 
on the Appointment Interviewer Worksheet. (See 
Appendix, page 42.) This process, it was hoped, 
would not only supply an indication of the number 
of therapy sessions, but would also provide a 
check on the study's requirement that a client be 
seen for no more than two intake interviews. (Ver- 
ification on the number of each client's intake 
interviews was done bimonthly with ten randomly 
selected clients assigned to treatment via the In- 
take Committee. For those ten clients, an audit 
was done from the Mental Health Service chart to 
insure that the client was seen for not more than 
two intake interviews before being assigned to treat- 
ment. For all cases in the bimonthly audits, the lim- 
it of two intake interviews wds obset'ved 100 percent.) 

According to the procedures begun in February, 
1970 once client visit information was recorded, 
all of the Appointment Interviewer Worksheets were 
taken to the Intake Committee meeting (which 
formally assigned to treatment clients requesting 
services). Having the worksheets already filled 
out with the client name and hospital number 
was done to save time. However, due to the 
large number ot clients who registered for ser- 
vices at the Mental Health Service but were 
never assigned to treatment via the Intake Com- 
mittee (i.e. , were terminated after intake or 
referred to another agency) , the process of re- 
viewing the Transaction Records was repetitious 
and time-consuming. 

All clients due for follow-up in a given month 
were recherked via their Mental Health Service 
charts to determine the actual number of times 
they were seen by their assigned therapist and 
whether or not they were still in treatment. It 
became clear that the onerous exercise of col- 
lecting and reviewing the transaction record was 
a redundant effort and was therefore discontinued. 
From then on, once the client was assigned, the 
Appointment Interviewer Worksheet was filled in 
at the Intake Committee meeting and became a 
permanent part of the client's Program Evaluation 
Project record. 



2. Determining Client Eligibility for Follow-up 
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3. Consent Form 

A third requirement for eligibility in the fol- 
low-up study was a signed consent form. (See Ap- 
pendi:'., page 41.) The conseri form was present- 
ed to the client at the time of registration for 
services. The Mental Health Service admissions 
clerk explained the study briefly and asked the 
client to cooperate with the follow-up interview. 
Since the client's contact with the admission clerk 
was usually before he/she had been seen for in- 
take, a number o^ clients were anxious and hesitant 
to sign anything. If the consent form was not 
signed dur^'ng the registration process, a consent 
form was sent to whichever therapist the client 
was later assigned, requesting that the therapist 
try once again to elicit the client's cooperation 
with the follow-up interview. It wa^ left to the 
therapist's own judgment to decide when it would 
be appropriate to again explain the research 
study and request the client's signature. If the 
therapist had not f.een the client, or if the 
client was st'll reluctant to sign, the blank f^rm 
was returned to the research office two months after 
the time the client had been assigned to treatment. 

If the consent form still wasn't signed at the 
time research staff were to contact the client fnr 
the follow-up interview, research staff then at- 
tempted to obtain verbal consent. Those agree- 
ing were presented a copy of the form tc sign at 
the outset of the follow-up interview. No client 
who had not signed the consent form or who had 
not verbally agreed to participate with the fol- 
low-up interview was approached by a follow-up 
interviewer. Of the clients who ultimately re- 
fused to participate in the interview, 24 had 
not signed the consent form at admission and 42 
had initially signed consent forms, but refused 
to cooperate with the interview at the time of 
fol low-up. 



homes. Some clients were followed up at Institu- 
tions such as Anoka State Hospital, the state 
prison, or the Hazel ton Treatment Center, all of 
which are outside of the Minneapol 1s-3t. Paul 
metropolitan area. 



5. Locating Clients 

One characteristic indicative of many clients 
seen at the Hennepin County Mental Health Service 
is transience. The follow-up dat.e for clients 
could be set at anywhere from two months to two 
years from the date of assignment. For most cli- 
ents, however, the date of follow-up was six 
months arter date of assignment; If th2 Intake 
interviewer Indicated no preference for date of 
follow-up, it was automatically assumed to be six 
months. Even in six months, many clients were 
unlocatable at the address and phone number listea 
at the time of intake. 

Most of the follow-up interview appointments 
for the four mode study were arranged with the 
client by a research staff member via telephone. 
For the interdisciplinary reliability study (Sae 
Section II of this report.) and other such studies, 
the interviewe^^s were asked to arrange their own 
appointmerits. There were some cases in which 
interviewers made an impromptu visit to the home 
of clients who had no phone but who had previously 
signed the consent form. 

If the client could not be reached at the phone 
number indicated at intake, a letter was sent from 
the research office requesting that the client phone 
the research o^^fice to schedule an appointment. If 
this letter came back to the office because the 
client was no longer at the address, the next course 
of action was to implement one or more of the follow- 
ing seven measures employed by the research staff to 
locate cl ients. 



4. Scheduling the Follow-Up Interviews 

The follow-up interviewers prepared a monthly 
schedule of the times they were available for 
follow-up interviews and also an estimate of the 
number of interviews they could attempt that month. 
This information was useful to the research staff 
for predicting any need to increase the inter- 
viewing staff to accommodate the number of clients 
due for follow-up in the subsequent months. 

Originally it was recommended that clients be 
randomly assigned to follow-up interviewers to 
prevent any assionment bias and to insure a bal- 
anced work load for all interviewers. This plan 
was ultimately abandoned since the follow-up in- 
terviewers were not reimbursed for mileage (un- 
less mileage for one interview exceeded 50 miles, 
round tr*Jp) ^nd preferred to work near their 
homes. Consequently, an attempt was made to 
minimize travel for the interviewers and random 
assignment to interviewer was not employed after 
approximately the first 100 follow-up interviews 
for the I our mode study. 

It was not always the case, however, that in- 
terviewers saw only clients who lived near their 

1 



a. If the client was still in treatment at 
the Mental Health Service, his therapist was con- 
tacted in case he might know of the client s ad- 
dress change. Therapists on occasion were re - 
quested to encourage the client to call the re- 
search office to schedule the appointment. 



b. A certified, registered letter was sent to 
the client's last known address. Certified mail 
is expensive (about $1.30 per letter) but proved 
to be helpful on many occasions. 



c. Because the Project staff did not have a copy 
of the City Directory, which, unlike the telephone 
book, lists everyone living in the city, a liaison 
arrangement was set up with the city electric and 
aas companies. They listed their customers both 
by name and address. The Program Evaluation Proj- 
ect contacted the designated liaison for the util- 
ity company and requested a check for verification 
of address or phone number. This effort was soon 
abandoned because the utilities were unable to be 
of much help. Their directories listed only cus- 
tomers, and for the most part, clients utilizing 
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the Mental Health Service facilities did not own 
their own homes and often were not responsible 
for paying heating or electric bills. Also, in 
the case of multiple dwelling establishments, the 
owner listed by the utility company was very often 
a corporation rather than an individual landlord. 
Even if the utility companies could locate the 
client on their records, they often times listed 
the same address as the Program Evaluation 'Proj- 
ect records, or the client had already left the 
recorded address with no forwarding address in- 
dicated. 



d. At the time of registration for services 
at the Mental Health Service, the clients were 
asked to list their nearest relative or friend 
who would probably know of their whereabouts in 
six months. This often was helpful, but it was 
surprising to note the number of parents, for 
example, who did not know the whereabouts of 
their children or were unwilling to disclose the 
information. All contacts made with persons other 
than the c»ient himself were carried out without 
violating the client's right to confidentiality 
regarding his contact with Mental Health Service. 
Relatives or friends were told that the Outpa- 
tient Department of General Hospital was trying 
to locate the client, and even this information 
was offered only if the inforrriant requested it. 
This same confidential approach was used while 
contacting employers or previous employers. 



e. If there was any indication in the cli- 
ent's Mental Health Service chart that he/she was 
receiving assistance from a public agency such 
as Welfare, Vocational Rehabilitation, etcetera, 
these agencies were contacted for address and phone 
information. Of these, the A.F.D.C. files and the 
Welfare master files were of most help. 



f. Some clients were involved with legal auth- 
orities at the time of intake (as noted on the Goal 
Attainment Follow-up Guide), which indicated the 
possibility of incarceration at the time of fol- 
low-up, a situation which made it difficult to con- 
tact the client directly. There was also on oc- 
casion an indication of a corrections officer in- 
volved with the client who, when contacted, was 
helpful in many cases. In a few instances the 
officers were able to arrange for a fol low-up in- 
terview to be conducted at the county workhouse, 
city jail, state reformatory, or state prison. 
Both the liaison with the utility companies and 
with the Bureau of Investigation were arranged by 
the follow-up coordinator for the research proj- 
ect. Attempts at obtaining information from 
local draft boards were made on two occasions, 
but because of the Boards* confidentiality poli- 
cies, both contacts proved to be extremely time- 
consuming and complicated, and though ultimately 
helpful, they preferred that the staff not contact 
them for this type of help. 



g. On several occasions, clients were located 
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out of sUte, If they agreed to participate in an 
interview by phone, the follow-up Interviewers 
placed a person-to-persor call to them and were 
reimbursed by the Prnjact for phone expenses. One 
attempt was made via letter to reach a client 
l"'ving in Israel, but this endeavor proved to be 
unsuccessful. Protection of the client's right to 
confidentiality was treated as a serious matter 
throughout the follow-up study and though the 
research staff's efforts to locate clisnts were 
extensive, no client ever reported being offended 
in any way. 

For purposes of recording contact efforts made 
by the research staff and to have an ongoing record 
of interactions with the client regarding the fol- 
low-up interview, a Follow-up Appointment Interview 
Worksheet (see Appendix, page 42) was used to 
record all attempts to find each client. The work- 
sheet also provided Information regarding when 
follow-up interviewers were available as well as a 
notation of the confirmed appointment date and 
time. Once the client was either followed up or 
eliminated from the study, the worksheet became 
part of the client's Program Evaluation Project 
chart. 

Most interviews were conducted in the client's 
home. If this was inconvenient or if the client 
desired greater confidentiality, the interview was 
scheduled at the Mental Health Service, the Pro- 
gram Evaluation Project office, or In a few car>es, 
a public establishment such as a restaurant. A.^^ 
mentioned previously, for the initial contact with 
the client (to arrange the follow-up interview) more 
than one phone call was usually required before the 
client was successfully located. 



F. Post-Interview Procedures 

Most of the follow-up unit's time was spent in 
arranging interviews. Important to the research 
endeavor, however, was the assimilation of the data 
collected by the follow-up interviewers during their 
encounter with the client. 

All follow-up packets were sent to the inter- 
viewer with a return envelope included. As soon 
after the interview as possible, the packet was 
scored by the intfc*"viewer and returned to the 
Program Evaluation Project office. The packet was 
reviewed by the follow-up supervisor and assistant 
supervisor to insure that all questions were an- 
swered, and that the Goal Attainment Follow-up 
Guide was scored. If the interviewer indicated 
any problems with either scoring che follow-up 
guide or answering any of the questions, the fol- 
low-up supervisor provided clarification. 



1. Special Scorings: Collateral Interviews 

Oftentimes some of the scales on the follow-up 
guides required checking with a source other than 
the client for verification of scoring, such as a 
collateral check with the client's corrections of- 
ficer to verify how many times, if any, the client 
violated the regulations of his probation. This 
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kind of check was completed by the research iitaff 
after the completion of the follow-up interview. 

Another common verification source for the 
scoring of a scale was the therapisti the 
number of times the client was actually see!i 
for therapy and whether the client discontinued 
therapy with the therapist's consent or tnerely 
stopped coming to the Mental Health Se'^vice. 
This check was also completed by the research 
staff since it was preferable that che follow- 
up interviewer remain unaware of che client's 
therapist or therapy mode. 

Another verification check that was sometimes 
required was the report of the client's spouse 
or employer. This type of check was completed by 
the follow-up interviewer, if the client agreed 
to the contp.ct by signing the Collateral Consent 
Form. (See Appendix, page 43.) 



2. Special Scorings: Tests 

Some scales also required the client to com- 
plete the Minnesota Multiphasic Personality In- 
ventory at the time of follow-up. In the cases 
for which the Minnesota Multiphasic Personality 
Inventory was required, it was sent to the follow- 
up interviewer with the follow-up packet, and the 
interviewer requested that the client complete it 
as soon as possible after the interview and re- 
turn it to the research office. A note was made 
in the client's Mental Health Service chart that 
the Minnesota Multiphasic Personality Inventory 
had been completed. The follow-up Minnesota 
Multiphasic Personality Inventory was a perma- 
nent part of the client^s Program Evaluation Pro- 
ject record and a copy of it was provided to the 
therapist or any other clinician upon request. 

In a few cases, the Mental Status Exam was ad- 
ministered at the time of follow-up and for these 
cases an M.S.W. follow-up interviewer who was fa- 
miliar with the exam, was assigned the follow-up 
interview. For one client the Weschsler Adult 
Intelligence Scale subtest 5, "Memory Digit Span" 
was required at the time of follow-up. 

In the case of any scales which were unscorable 
at the time of follow-up, ( e.g. , the client refused 
to complete the Minnesota Multiphasic Personality 
Inventory, or refused to discuss certain problem 
areas), the scales were eliminated from the follow- 
up guide and not used in the comput?^ion of the 
Goal Attainment score. Some scales were inap- 
propriate at the time of follow-up if they referred, 
for example, to educational progress and the client 
was not in school but employed full-time instead. 
These scales, if unscorable according to the inter- 
viewer, were also eliminated from the Goal Attain- 
ment score. Among the 708 follow-up cases including 
2507 scales, only 81 scales (3.2%) were considered 
unscorable. 

Once the packet was reviewed and the scales 
scored, the completed follow-up was noted and the 
packet placed in the Program Evaluation Project 
data fi'le. Soon after the completed interview. 



the follow-up packet was coded (See chapter on four 
mode study procedures in the P.E.P. Report. 1969- 
1973.) for keypunching and submitted for inclusion 
orTthe Program Evaluation Project computer tape 
file (See chapter on Computer Information Systems 
in the P>E.P. Report, 1969-1973 .) 



3. Clients Eliminated from Follow-Up 

Clients were eliminated for a variety of rea- 
sons, as indicated on the Elimination Code Sheet. 
(See Appendix, page 44.) Only part of the group 
of clients non-randomly assigned to treatment were 
followed-up, because of budgetory limits. Those 
followed-up of this group were selected at random. 
Clients who refused to participate in the follow- 
up interview were also eliminated, as well as those 
who could not be located by the research staff at 
the time of follow-up. 

It was further determined that the latest a 
client could be interviewed was six months after 
the follow-up date prescribed by the intake inter- 
viewer, because the goal 3 were established to be 
appropriate at a specific follow-up date. It was 
quite common, however, not to be able to locate a 
client until well after the prescribed follow-up 
date. 

Inability to locate was the most frequent 
reason for elimination. Of the 2,096 clients for 
whom a Goal Attainmejit Follow-up Guide had been 
completed, 1,388 were eliminated from follow-up 
for various reasons. Two hundred and twenty-three 
(10.5% of the clients with follow-up guides) 
could not be located by the research staff at the 
time of follow-up. 

A client was elimina*'^d from the follow-up 
study as an assumed refus r if he failed to keep 
three consecutive scheduled follow-up appointments. 
Approximately 15 percent of all scheduled appoint- 
ments for a given month were failed by the client 
on the first scheduling. Of the 1,388 clients 
eliminated from follow-up, 66 (4.32%) were consid- 
ered refusals. (See Table II on following page.) 

Originally the elimination of clients was de- 
termined by an elimination corrmittee consisting of 
the Project's assistant director, follow-up super- 
visor, and assistant supervisor. This committee 
was dissolved at the end of the first year of 
active follow-ups. The elimination decision was 
then made bv the research staff member responsible 
for attempting to locate and contact the clients, 
who was best acquainted with the case and thus 
able to determine the reason for elimination. 



G. Estimates of Time and Cost Expenditures for 
the Follow-Up Unit 

In December, 1971, the research staff initiated 
a time study to determine the approximate staff 
time invested in foi low-up. The audit involved 
record keeping of time spent on 1) determining 
eligibility for the 100 clients "potentially" eligi- 
ble for that month, 2) locating the 33 clients se- 
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TABLE II: Elimination from Follow-Up 
(N-13dd) 



^ea:>ons for elimination number 

eliminated 



Goal Attainment Fol low-Up Guide 
Follow-up guide completed after 



client entered treatment 


2 


Client previously included in 
study and followed up 


2 


Invalid follow-up guide 


16 


Course of Therapy 




Zero therapy sessions completed in 
assigned therapy mode 


216 


One therapy session completed and 
client terminated f^'cm treatment 


205 


Other 


2 


CI lent consent 




Client refuses to sign Program 
Evaluation Project consent 


24 


Client has signed consent but re- 
fuses intervi^ at time of follow-up 


39 


Other 


3 


Client Cannot Be Contacted 




Client deceased a* Lime of follow-up 


5 


Client has moved out of state 


27 


Client has moved out of metro area 
and Is not reachable 


20 


Program Evaluation Project unable to 
locate client 


171 


Low Priority for Follow-Up 




Random assignment to major modes 


10 


Honrandom assignment to major modes 


252 


Nonrandom assignment to minor modes 


202 


Assignment to termination 


181 


Other 




Random client eligible for follow-up 
but eliminated on the basis of cost 


4 


Followed up but eliminated dw' to 
unscorable follow-up guide 


2 


Other 


5 



!ected for follow-up, 3) preparing the packets for 
mailing, 4) informing the interviewer of the 
scheduled appointment. 5) mailing the packets, and 
6) review of the completed interview packets . 

Twenty hours were required to determine that 33 
of the 100 clients had completed a minimum of two 
therapy sessions. The average number of contacts 
needed to successfully locate the client to arrange 
an interview was five. Such attempted contacts in- 
cluded phone calls, letters, and discussions with 
efnployers, welfare and so on. The mean time spent 

1 



in locating each client was 25 minutes. Prepara- 
tion of the interview schedule. Including dup- 
lication of the Goal Attainment Follow-up Guide 
and preparing it for mailing, also required 25 
minutes per client. Only five minutes was spent 
contacting the follow-up interviewer to confirm 
the appointment for each interview. For each cli- 
ent eligible for follow-up, approximately one hour 
was needed tc arrange thn appointment and prepare 
the follow-up packet for the Interview. When a 
client failed a scheduled appointment, the Inter- 
velwer contacted the research office and an at- 
tempt was made to reschedule the appointment. The 
above estimation of' time expenditure Includes any 
rescheduling nvc'?ssary to complete the follow-up 
interviews. 



1. Cost Figures, Follow-Up Interviewers 

The Program Evaluation Project was involved in 
follow-up interviewing from June, 1970 through the 
end of November, 1973. As mentioned previously, 
a variety of interviewers were employed through 
the years of the project, with varying salary rates. 
The following figures, based on annual expendi- 
tures, consistently Include reimbursements paid to 
the follow-up interviewers for both completed and 
failed interviews and are included here as hope- 
fully relevant for budgeting estimations for sim- 
ilar follow-up progrsms, given the types of in- 
terviewers employed by the Program Evaluation 
Project and the number of interviews completed 
over the past four years. 

During the months of June through December, 
1970, 107 interviews, were completed. The total 
cost for interviewing time was $1,915 and during 
this time, only M.S.W.s were employed as follow- 
up interviewers. Mean cost per completion was 
$17.90. 

Mid-year, 1971, bachelor-degreed nurses and 
teachers' aides were added to the interviewing 
staff at a rate of payment less than J, at of the 
M.S.W.s. Also, in 1971, the follow-up unit be- 
gan conducting follow-up Interviews for the 
Crisis Intervention Center of Hennepin County 
General Hospital, after the Center Implemented 
Goal At' inment Scaling in July. (See chapter 
on the C isis Intervention Center in P.E.P. Report , 
1969-1973. ) By the end of 1971, 38 follow-ups 
had been completed for the Crisis Intervention 
Center at a total cost of $690, or 9.66 percent 
of the year's total expenditures for interviewing 
time of $7,143.46. The total number of follow- 
up interviews completed for the Mental Health 
Service was 420, with a resulting cost of 
$6,453.46 or 90.34 percent of the year's total. 
The mean cost per completed interview was $19.21 
for the Crisis Intervention Center, $16.37 for 
the Mental Health Service, and $15.60 overall - 

The total cost of follow-up interviews for 
1972 was $5,677. Added to the aforementioned 
follow-up staff were B.A. social workers and under- 
graduate students from the University of Minnesota. 
Telephone interviews were initiated late in 1971 
and continued throughout 1972. The M.S.W.s re- 
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ceived $10 for completed phone interviews and all 
other interviewers received $6. resulting in an 
expense decrease. No payment was made for failed 
phone interviews. 

The total cost of the 299 interviews completed 
for the Mental Health Service was $3,897.50, or 
68.65 percent of the total follow-up expenses in 
1972 (mean cost per interview $13.04) and for the 
Crisis Intervention Center's 57 follow-ups, $671. 
50 or 11.83 percent of the 1972 total. 

In March of the same year, follow-up began for 
the drug effectiveness study comparing Valium and 
psychotherapy, a joint venture of the staffs of the 
Mental Health Service and the Program Evaluation 
Project. Because of the nature of the study, only 
the experienced (those hired prior to March of 
1971) interviewers were to conduct the follow-up 
interviews. Each client included in the study 
required three separate follow-up interviews at 
three weeks, two months, and six months after be- 
ginning treatment. Payment for completion of the 
drug study follow-ups was set at $17.50 because 
of the increased time involvement necessary for 
the client and the interviewer to complete the 
Self-Rating Symptom Scale and the Brief Psychiatric 
Rating Scale respectively at the end of the inter- 
view. During the months of March through December, 
56 drug study follow-ups were completed at a total 
cost of $1,108 or 19.52 percent of the expendi- 
tures for interviewing time for 1972, at a mean 
cost of S19.78 per interview. 

The Project ceased data col' action for the 
four mode study in November of 1972 with the 
advent of the Mental Health Service re-design 
program the same month. (For details, see 
chapter nine in the P.E.P. Report 1969-1973 .) 
Thus, follow-up expenditures for 1973 were 
significantly less than the previous years. 
The drug study continued through 1973 with the 
completion of 21 interviews at a resulting cost 
of $352.80 or 36 percent of the year's total 
expenditures for interviewing time of $1056.50. 
The final 33 interviews for the four mode study 
cost $352.80 to complete or 34 percent of the 
year's total. In February the Program Evaluation 
Project began follow-up for the Guide to Goals 
study (See chapter on varieties of Goal Attain- 
ment Scaling in the P.E.P. Report 1969-1973 .) 
for the Hennepin County Day Treatment Program 
and completed 34 interviews for a total cost 
of $321.20 or 30 percent of the year's total. 



2. Summary 

During the four years of the Program Evaluation 
Project, 1128 follow-up interviews were completed 
totaling $15,791.98 with the average cost per in- 
terview of $13.99. The mean number of interviews 
completed per year was 282 with a mean of $3,948 
expended per year for follow-up interviewers' 
salaries . 



ganizing and implementing the follow-up study for 
the Program Evaluation Project. In mid-1973, the 
follow-up unit asked some of the follow-up inter- 
viewing staff for consents and recommendations for 
follow-up. The following were offered by follow- 
up interviewers experienced with scoring the Goal 
Attainment Follow-up Guides and the author. 



1 . Confidentiality 

It is important to keep in mind a client's 
rights with regard to confidentiality of services 
received, especially in the area of mental health. 
While attempting to locate clients to arrange a 
follow-up interview, the research staff, and in 
some cases the follow-up interviewers, always 
identified themselves to the client, but when talk- 
ing to other persons (i.e., employers, relatives, 
friends, and so forth), the identification aivj rea- 
son for calling were handled as discreetly as pos- 
sible. Letters were often sent to the client's 
last known address if they were unlocatable by 
phone, and in all cases clients for whom an appoint- 
ment had been scheduled received a letter of re- 
minder a day or two prior to the interview. All 
correspondence was sent without a return address 
identifying either the Program Evaluation Project 
or the Mental Health Service. At times this dis- 
creetness turned into fol iy as in the case of a 
certified letter sent to one client who assumed 
the follow-up interviewer to be a bill collector 
and avoided responding to either phone calls or 
letters. In due time, however, the misunder- 
standing was cleared up and the interview completed 
as prescribed. 



2. Privacy 

Closely related to the area of confidentiality 
is the need for privacy in conducting the actual 
interview. This, above all else, was the recom- 
mendation of the follow-up interviewers. One in- 
terviewer commented that he had conducted inter; 
views in cars, basements, fron* steps, gas stations, 
bars, back yards, and offices, to name a few, and 
preferred all of these to a lack of privacy. (Bes- 
nett, 1973) This recorrmendation is mostly for the 
protection of the client, but also relates to ac- 
curate data collection in lieu of a client's pos- 
sible unwillingness to be candid if the interview 
is being witnessed by friends, parents, children, 
or a spouse. 

Though the phone interviews completed during 
the course of the Project resulted in no signifi- 
cantly different Goal Attainment scores than those 
scores obtained through in-person interviews (See 
Section IIB of this report.), follow-up interview- 
ers felt clients were more guarded in their re- 
sponses. Phone interviews, though less expensive 
to complete, are not always conveniently private 
for the client, and may affect the client's open- 
ness and willingness to discuss problem areas de- 
lineated by the Goal Attainment Follow-up Guide. 



H. Recofimendations 

A great deal of experience has been accumula- 
^•ed by the research staff during the course of or- 
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3. Quality of Follow-Up Guide Construction 

A system for assessing the utility of the Goal 
Attainment Follow-up Guide was completed by Gar- 
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wick and Grygelko (See chapter on assessment of 
the Goal Attainment Fonow-up Guide, P>E>P> Report 
1969-1 973 >) and discussed elsewhere but it is ap- 
propriate to mention here one of the most frequent 
difficulties follow-up interviewers had to cojDe 
with while scoring the Goal Attainment Follow-up 
Guide* Though the research staff did attempt to 
clarify the content of follow-up guides in nego- 
tiations with the follow-up guide constructors, 
the mixing of two or more variables per scale 
level made some scales impossible to score at the 
time of follow-up if the two behaviors did not 
vary together over time» 

A suggestion for alleviation of the problem 
would be to employ follow-up guide constructors 
as follow-up interviewers. If at all possible* 
The same experience could be realized from role 
plaiying or other techniques that would involve 
the foTlow-up guide constructors in the role of 
follow-up interviewer (i»e», knowing nothing 
about the client except perhaps age, sex, marital 
status, and those areas delineated by the Goal 
Attainment Follow-up Guide). This experience of 
role reversal would, it is hoped, emphasize the 
desirability of clear, succinct, and behaviorably 
observable statements for inclusion on the Goal 
Attainment Follow-up Guides. 



4. Training of Follow-up Interviewers 

Familiarizing the follow-up interviewers with 
Goal Attainment Scaling is necessary to insure 
accurate scoring of the Goal Attainment Follow-up 
Guides. An interviewer procedures manual (Audette 
and Garwick, 1973) was developed which delineates 
the rudiments of accurate scoring of the Goal At- 
tainment Follow-up Guides. One of the primary 
requests made of the interviewers was that they 
score the Goal Attainment Follow-up Guide inde- 
pendently, without collaboration with fellow follow- 
up interviewers. Because data collected for this 
study was based heavily on foliow-up interviewing, 
it is important that the interviewers be prepared 
to elicit accurate information without alienating 
the client during the interview. Interviewers 
were paid for their attendance at training sem- 
inars, and that cost was incorporated into the 
annual mean cost per interview reported above. 

Though it is true that good follow-up data is 
dependent upon well constructed follow-up guides, 
it is also important to remember that well trained 
follow-up interviewers are a key factor in the 
collection of outcome data for any follow-up study. 



II. Special Studies Undertaken by the Follow-ug^ 
Unit 



The major part of the follow-up program from 
1969 to 1973 was devoted to the collection of data 
for the original study comparing four modes of men- 
tal health treatment. However, the follow-up unit 
was involved with a number of other smaller studies 
each of which supplied additional information about 



the follow-up process. These are discussed in the 
following section. (In this section, correlation 
coefficients and differences in means are consid- 
ered to be not statistically significant unless 
they ror.,.f' th'j p <.05 level.) 



A. The Original Reliability Study 

1. History of the Original Reliability Stucty 

A requirement of the original National Insti- 
tute of Mental Health funding to the Program Eval- 
uation Project was that the research staff complete 
two "reliability studies" concurrently with the 
main evaluation at the Mental Health Service. 

These two largely separate reliability studies 
were combined into one overall research design in 
such a manner that both studies could be carried 
out simultaneously. (They, are discussed in de- 
tail in the chapters on reliability in the P.E.P . 
Report, 1969-1973. ) 

a. The Fol low-Up Guide Construction Re- 
liability Study 

The purpose of this studiy was to measure the 
consistency (as measured by the degree of similar- 
ity in Goal Attainment scores obtained at the time 
of follow-up) between two follow-up guide con- 
structors (a client's intake interviewer and his 
assigned therapist) who independently constructed 
follow-up guides for the same client at approx- 
imately the same point in time. 



b. The Fol low-Up Reliability Study 

This study was designed to measure the con- 
sistency (again as measured by the Goal Attain- 
ment scores) of two follow-up interviewers, who 
independently but at approximately the same point 
in time, scored the follow-up guides constructed 
for a particular client. 



2. Implementation of the Study 

The study began in May, 1970. Fifty clients 
were to be included in the study, 25 randomly as- 
signed to treatment and 25 nonrandomly as^iigned 
to treatment. It was assumed that there would be 
some attrition from the desired number of 50 and 
that complete data on at least 40 clients would 
be sufficient for the completion of the study. 

The reliability study was officially closed on 
October 1, 1972, with the completion of 44 dual 
follow-up interviews (26 random and 18 nonrandom 
assignments to treatment). Throughout the study, 
intake and therapist Goal Attainment Follow-up 
Guides were requested for 164 clients. Of these, 
a total of 84 subjects were actually accepted into 
the study. Table III reports the reason for elim- 
ination of the 40 clients not included in^the 
sample. (See chapters on reliability in the P.E.P . 
Report, 1969-1973. ) 
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TABLE III: fMminations frorr, Original Rel iabil i ty Study 



UNLOCATABLE 



3 REFUSED 



FIRST INTERVIEW SECOND INTERVIEW 



12 



ADMINISTRATIVE 
REASONS, e.g. 
Invalid Toll^w-jp 
Guide 



3. Sunmary of the Data 

The mean number of days between the first and 
second interview was 25 and in no case was the 
first and second interview conducted by the same 
follow-up interviewer. The greatest number of 
days between the two interviews was 67 and the 
shortest time elapsed between the first and 
second interviews was five days. 

The Pearson Product Moment correlation co- 
efficient was employed to determine the relation- 
ship between the difference of the Goal Attain- 
ment score for the first and second interviews, 
and the length of time (in days) between first 
and second interview. Two separate correlations 



were derived: 1) for the difference in Goal At- 
tainment scores for the intake Goal Attainment 
Follow-up Guides, which equals -.193, and 2) for 
the difference in scores for the therapist Goal 
Attainment Follow-up Guides, which equals .059. 
The assumption drawn from the results of this com- 
putation IS that the length of time between the 
first and second follow-up interviews had no sig- 
nificant effect on the difference of th"" Goal At- 
tainment scores from the two interviews. 

Table IV reports the mean and standard de- 
viations for the Goal Attainment scores from the 
intake Goal Attainment Follow-up Guides and thera- 
pist Goal Attainment Follow-up Guides for the 
first and second interviews. Computation of the 



TABLE IV: Mean Goal Attainment Scores for Both Follow-ups and Both Guides, 
By Number of Days Between FoUow-Ups 



Number of Days between First 
and Second Interview: 



- 14 



15 - 29 



30 - 44 



45 - 67 



TOTAL 



Number of Subjects 



N«12 N»17 N«ll N-4 



FIRST INTERVIEW 



In .e Interviewer Mean Goal 
Attainment Score 



Therapist Mean Goal At- 
tainment Score 



S^rsm INTERVIEW 



. X^ke. Interviewer Mean Goal 
Atuinment Score 



Therapist Mean Goal At- 
tainment Score 



44.68 



49.04 



N = 44 



51.87 



55.68 



45,55 50.97 47.61 50.62 



48.13 55.56 47.53 54.77 



51.99 



54.91 



50.63 



54.57 



48.62 
S.D.=9.18 

51.45 
S.D.=9.84 



49.83 
S.D.=11 .18 

53.57 
S.D.=8.89 



O 
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two tailer T-test yielded no significant differences 
'-etween Ine Goal Attainment scores for first and 
second interviews. (See Table V.) 

TABLE V: Goal Attainment Score Analysis for 
First and Second Interviews 





iHlf^KL UJID: 


THul^'Iiir GUIOE 


OVERALL 


FIRST INTERVIEW 


N ' 14 


N • 44 


N - 88 




X « 48.62 


X - 51 .45 


X - 50.04 




S.O. " S.18 


S.O. ■ 9.84 


S.O. ■ 9.57 




S.C. • 1.38 


S.E. • 1.48 


S.E. - 1.02 


SECOND INTERVIEW 




N 44 


N - 88 




i • 49.83 


J - 53.57 


X • 51.70 




S.O. ■ 11.18 


S.n. -- 8.90 


S.O. - 10.22 




S.£. - 1.69 


S.E. - 1.34 


S.E. ■ ;.09 


OVERALL MEAN 
OF FIRST AhD 
SECOND 
INTERVIEWS 


N ■ 88 
i ■ 49.22 
S.O. - 10.19 


N - 88 
X - 52.51 
S.O. ■ 9.39 


N - 176 
X • £0.87 
S.O. ■ 9.91 




S.E. - 1.09 


S.E. • 1.00 


S.E. ■ .75 
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Figures H and III report the distribution of 
Goal Attainmeiit scores from the first and second 
follow-up interviews, intake and therapist con- 
structed follow-up guides. (See next page.) 



4. Consumer Satisfaction 

Table VI reports the consumer satisfaction ini- 
formation collected during the first and second in- 
terviews. At the outset of the study it was assumed 
that it was not necessary to collect consumer satis- 
faction information during the second interview. 
The decision was reversed once the consumer satis- 
faction questions were deemed more valuable for data 
comparisons. Unfortunately, not all clients, dur- 
ing the second interview, were asked to respond to 
the consumer satisfaction questions. The dual fol- 
low-up interview procedure did not negatively af- 
fect the clients* evaluation of services received 
while at the Mental Health Service as can be seen 
in Table VI. 



TABLE VI: Consumer Satisfaction Results for the Clients in the Original Reliability Study 



1. How would you describe your feelings about seeking service at the Mental Health Service? 



Very positive. Positive. 

very willing sonewnat willing 



Indifferent Keoatlve. sooewhat Very negative. 

anblvalent very ambivalent 



First 

Interview 9 20.45X 


23 52.27X 


2 4.54t 


9 2*3. 4SX 


1 2.27X 




Second 

Interview 14 31 .821 


22 50.00X 


2 4.S4X 


2 4.S4S 




4 9.091 


Did you have any problems 9ettlng service at the Mental Heal 
No Yes No answer 


viceV 


First 

Interview 35 79.541 


9 20.45X 




Second 

Interview 34 77.27X 


6 13.641 


4 9.091 



3. How satisfied were yoo with the services you received? 

Very satisfied Satisfied Indifferent Dissatisfied 

First 



Very dissatisfied No answer 



Second 



17 38.641 


13 6.B2t 


4 9.091 


9 20.45X 


1 2.271 




13 29.54X 


14 31.82X 


7 15.911 


5 11.36X 


1 ?.27l 


4 Q noi 



4. Do you feel differently about your problens now? 
Much better Better 

First 

Intervli 

Second 



18 40.91X 


21 47.731 


5 11.361 








16 36.36X 


24 54.541 


4 9.091 









5. Do you attribute this change or lack of change to trotfflcnt at the Mental Health Service? 

Yes. mostly Yes, partly Not for «'jst Not at all No answer 

part 



first 

Interview 20 




13 


29.54X 


7 15.901 


3 


6.82X 


1 2.27;^ 


Second 

Interview 16 


36.361 


16 


36.361 


6 13.631 


5 


11.361 


1 2.27X 
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INTAKE INTERVIEWER CONSTRUCTED FOLLOW-UP GUIDE 



NUMBER 
OF 

CASES 



NUMBER 
OF 



CASES 



10 
9 



26 28 10 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 73 69 

GOAL AHAINMENT SCORES 

THERAPIST CONSTRUCTED FOLLOW-UP GUIDE 



FIGURE II: Distribution 
of Goal Attainment Scores, 
Intake and Theiaplst 
Follow-up Guides 

FIRST INTERVIEW 



10 



26, 28 30 32 34 36 3840 42 44 4g 48 50 52 54 56 53 60 62 64 66 68 70 72 74 76 78 

GOAL AHAIN.MENT SCORES " 



INTAKE INTERVIEWER CONSTRUCTED FOLLOW-UP GUIDE 



FIGURE III: Distribution 

of Goal Attainment Scores. NUMBER 

Intake and Therapist 

Follow-up Guides 



SECOND INTERVIEW 



CASES 



NUMBER 
OF 

CASES 




26 23 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 7B 80 

GOAL AHAINMENT SCORES " 



THERAPIST CONSTRUCTED FOLLOW-UP GUIDE 




26 23 30 32 34 26 38 40 42 44 46 48 ">0 52 54 56 58 60 62 64 £6 68 70 72 74 76 7a 80 

GOAL AHAINMENT SCORES 
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B. Interdisciplinary Reliability Study Using Phon e 
and In-Person Interviews 



1. History of the Study 

The purpose of this study was to determine 
scoring consistency between follow-up interviewers 
of different academic and professional backgrounds 
and, secondly, consistency between in-person and 
telephone interviews as measured by the Goal At- 
tainment score. Originally teachers' aides as 
well as social workers and nurses were to be in- 
cluded in the study, but it was decided that use 
of teachers' aides was not feasible. Consequently, 
the research design was modified to include only 
nurses and master's level social workers. 



3. Completion of the Study 

The original design was planned for a total of 
80 subjects to be interviewed twice, five in each 
of the 16 cells. In all, 141 clients were in- 
cluded in the study over a ten month period. Of 
these, 60 were interviewed twice and the remain- 
ing 81 subjects were eliminated from the study 
for the following reasons: 57 clients were un- 
locatable for either the first or second inter- 
view; 18 clients refused to cooperate v/ith either 
the first ov second interview, and 6 clients were 
eliminated for administrative reasons, e . g . , no 
access to telephone, deceased, or ir»yaiid fol- 
low-up guide. Table VH shows the distribution 
among the cells of the 60 subjects who partici- 
pated in the two follow-up interviews. 



2. Study Design 



4. Presentation of the Data 



I 
N 
T 
E 
R 
V 
I 
E 
W 



A randomly selected sample of clients who were 
nonrandomly assigned to treatment via the Mental 
Health Service intake committee and who had re- 
ceived a minin^um of two therapy sessions were in- 
cluded in the study. They were assigned by means 
of a random card deck to one of the 16 cells shown 
in Table VII. An attempt was made to interview 
each client twice, the type of interview and in- 
terviewer being determined by the cell to which 
the client v/as assigned. 

TABLE VII: Discipline of the Interviewers for the 
Interdisciplinary Reliability Study 

SECOND INTERVIEW 





RN in 

person 


RN 

phcne 


MSW in 
person 


MSW 
phone 


TOTAL 


RN-in 
person 


2 


3 


4 


6 


15 


RN 

phone 


3 


2 


4 


4 


13 


MSW-in 
person 


3 


2 


1 


2 


8 


MSW 
phone 


8 


6 


4 


6 

I 
i 


24 


Total 

1 


16 


13 


13 


1 

18 1 60 



For tills study, follow-up interviewers '..are 
asked to arrange their own appointments. Tne 
mean number of days between the first and second 
interviews was 27 and in no case was the first 
and second interview conducted by the same in- 
terviewer. 



The overall mean (combining first and second 
interviews) of the 52 in- person interviews is equal 
to 50.22, and the overall mean of the 68 phone in- 
terviews is equal to 52.90. The overall mean 
(combining first interview and second interview) 
of the 63 interviews, completed by the social work- 
ers is equal to 50.96, while the overall mean for 
the 57 interviews completed by the nurses is equal 
to 53.24. 

Tables VIII through XI present the means and 
standard deviation for mode of interview as well 
as types of interviewers across first and second 
follow-ups. Also included is the correlation co- 
efficients, (Pearson) and the results of the 
two-tailed T-test computed to determine signifi- 
cant differences in the means. None of the dif- 
ferences in means reached the p > .05 level of 
significance. 

The correlation coefficients for all combina- 
tions of interviewers were moderately high. The 
overall correlation of first inter» iew and second 
interview (N=60) is equal to .65, md the correla- 

on between in-person and phone interviews (N= 
31) is equal to .66. The correlation between 
scores derived by M.S.W.s for the first interview 
and M.S.W.s for the second interview (N=13) equals 
.61. The first interview by R.N. and second in- 
terview by R.N. (N-10) yielded a correlation co- 
efficient equal to .57. For first interviews 
completed by M.S.W.s and second interviews cofn- 
pleted by R.N.s (N=19) correlation of Goal At- 
tainment scores was equal to .59 For the first 
interview completed by R.N.s and the second in- 
terview completed by M.S.W.s (N=18), the cor- 
relation is equal to .80. 



5 . Conclusions 

The advantage of phone interviewing is that it 
is less expensive to complete, as was mentioned in 
Section I, part 4 of this report, and thus allows 
for the possibility of completing more interviews 
for less money. The information gathered from the 
interdisciplinary, phone versus in-person study 



21 



17 



ERIC 



TABLE VIII: Mean Goal Attainment Scores for First and Second Interviews 
According to Mode of Interview 





FIRST INTERVIEW 


SECOND INTERVIEW 


OVERALL MEAN 






IN PERSON 


N = 23; Mean = 47.78 
S.D. = 8.75 
S.E. = 1.82 


N = 29; Mean 52.66 
S.D. = 14.40 
S.E. = 2.67 


N = 52; Mean = 
S.D. = 2.36 
S.E. = 1.71 


50. 


22 


PHONE 


N = 37; Mean = 53.64 
S.D. = 12.65 
S.E. = 2.08 


N = 31 • Mean = 52 17 
S.D. = 10.89 
S.E. = 1.96 


N = 68- Mean = 
S.D. = 11.82 
S.E. = 1.43 


52 


90 


TOTAL 


N = 60; Mean = 50.71 
S.D. = 11.61 
S.E. = 1.50 


N = 60; Mean = 52.42 
S.D. = 12.60 
S.E. = 1.63 


N = 120; Mean = 
S.D. = 12.15 
S.E. = 1.11 


51 


.56 


T-Test Results: 












First Interview, phone vs. in person: p = > .01 
Second Interview, phone vs. in person: No significant difference 
In Person, first interview vs. second interview: No significant difference 
Phone, first interview vs. second interview: No significant difference 






Correlation: first interview and second 


interview = .646 









TABLE IX: Mean Goal Attainment Scores for First anc Second Interview According 
to Discipline of Interviewers 



FIRST INTERVIEW 


SECOND INTERVIEW 


OVERALL MEAN 


CORRELATION 


N = 32; Mean = 50.02 

SOCIAL 

WORKER S.D. = 12.59 
S.E. = 2.23 


N = 31 ; Mean = 51.36 

S.D. = 12.91 
S.E- = 2.32 


N = 63; Mean = 50.69 

S.D. = 12.66 
S.E. = 1.60 


.609 


N = 28; Mean = 52.96 

REGISTERED 

NURSE S.D. = 10.33 
S.E. = 1.95 


N = 29; Mean = 53.52 

S.D. - 12.39 
S.E. = 2.30 


N = 57; Mean = 53.24 

S.D. = 11.33 
S.E. = 1.50 


.571 


N = 60; Mean = 51.49 

TOTAL ^ ^ ^„ 
S.D. = 11 .59 

S.E. = 1.50 


N = 60; Mean = 52.44 

S.D. = 12.60 
S.E. = 1.63 


N = 120; Mean = 51.96 

S.D. = 12.06 
S.E. = 1.10 


.646 


T-TEST RESULTS: 

Social Worker, first interview vs. second interview: No significant difference 
Nurse, first interview vs. second interview: No significant difference 
First Interview, social worker vs. nurse: No significant difference 
Second Interview, social worker vs. nurse; No significant difference 
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TABLE X: Breakdown of Mean Goal Attainment Scores for Registered Nurses by Mode of 
Interview and Order of Interview 



FIRST INTERVIEW SECOND INTERVIEW OVERALL MEAN 


N = 15; Mean = 49.57 

IN 

PERSON S.D. = 8.22 
S.E. = 2.12 


N = 16; Mean = 53.09 

S.D. = 13.93 
S.E. = 3.48 


N = 31; Mean = 51.33 

S.D. = 11.48 
S.E. - 2.06 


N = 13; Mean = 56.87 

PHONE S.D. = 11.42 
S.E. = 3.17 


N = 13; Mean = 54.05 

S.D. = 10.73 
S.E. = 2.98 


N = 26; Mean = 55.46 

S.D. = 10.95 
S.E. = 2.15 


N = 28; Mean = 53.22 

TOTAL 

S.D. = 10.30 
S.E. = 1.95 


N = 29; Mean = 53.52 

S.D. = 12.39 
S.E. = 2.30 


N = 57; Mean = 53.39 

S.D. = 11.33 
S.E. = 1.50 


T-TEST RESULTS: 

In Person, first interview vs. second interview: No significant difference 
Phone, first interview vs. second interview: No significant difference 
First Interview, in person vs. phone: p ^ -01 
Second Interview, in person vs. phone: No significant difference 

Correlation: first interview, R.N. and second interview, R.N. = .571 


TABLE XI: Breakdown of Mean Goal Attainment Scores for Masters Degree 

Social Workers According to Mode of Interview and Order of Interview 


FIRST INTERVIEW SECOND INTERVIEW OVERALL MEAN 


N = 8; r n = 44.43 

IN 

PERSON S.D. = 9.25 
S.E. = 3.27 


N = 13; Mean = 52.12 

S.D. = 15.51 
S.E. = 4.30 


N = 21 ; Mean = 48.27 

S.D. = 13.74 
S.E. = 3.00 


N = 24; Mean = 51 .89 

PHONE S.D. = 13.16 
S.E. = 2.69 


N = 18; Mean = 50.81 

S.D. = 11.11 
S.E. = 2.62 


N = 42; Mean = 51.42 

S.D. = 12.19 
S.E. = 1.88 


N = 32; Mean = 48.16 

TOTAL 

S.D. = 12.59 
S.E. = 2.23 


N = 31 ; Mean = 51 .46 

S.D. = 12.91 
S.E. = 2.32 


N = 63: Mean = 49.81 

S.D. = 12.66 
S.E. = 1.60 


r-TEST RESULTS: 

In Person, first interview vs. second interview: No significant difference 
Phone, first interview vs. second interview: No significant difference 
First Interview, in person vs. phone: No significant difference 
Second Interview, in person vs. phone: No significant difference 

Correlation: first interview by social worker and second interview by social 
worker = .609 
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suggests that there 1s no significant difference in 
mean Goal Attainment scores between two types of 
interviews or between the two types of interviewers 
whose scorings correlate in the .59 to .80 range. 



Follow-up Study: Hennepin County Crisis In- 
terventi'ijn Center Comparing Professional alid 
Non-Professional Interviewers 



1 . History of the Study 



ign for the Crisis Interven- 
p study involved two main 
ng procedure of walk-in cli- 
n the stucty and, secondly, 
a Goal Attainment Follow-up 
fulfillment, wherein the 
nd the client negotiated 
therapy goals. (See chapter 
on Center of the P.E.P. Re- 



The research des 
tion Center follow-u 
procedures: a sampli 
ents for inclusion i 
the construction of 
Guide using contract 
intake interviewer a 
mutually acceptable 
on Crisis Interventi 
port, 1969-1973.) 

Every 15th walk-in client at the Crisis In- 
tervention Center had a Goal Attainment Follow-up 
Guide constructed for his case. In total 
303 guides were constructed and of these, 109 
were followed up. Prescribed follow-up time was 
three vveeks from the date of initial contact with 
the Center. Very few clients, however, were lo- 
cated before or on the suggested date of follow- 
up. This was due mostly to the transience of the 
population and also the fact that many clients 
had very short or one time contacts with the 
Center. The clients often left no permanent ad- 
dress or a temporary address which was no longer 
applicable at the time of follow-up. 



2. Implementation of Fol low-Up 

Three types of follow-up interviewers were em- 
ployed for the completion of this study, including 
the Program Evaluation Project's experienced 
master-level social workers, undergraduate univer- 
sity students, and registered nurses. The inter- 
views were conducted in person when possible, and 
via telephone only in cases when an in-person in- 
terview was not possible. Only one interview was 
conducted by an R.N., and of the remaining 108 
interviews completed, 31 or 28.44X were conducted 
by undergraduate students (juniors and seniors 
majoring In the social sciences), and 77 or 70.64% 
were completed by the M.S.W.s. If clients had 
more than one contact with the Crisis Intervention 
Center, they were seen for follow-up more than 
once. However, each interview evaluated a differ- 
ent set of treatment goals as indicated on sep- 
arate Goal Attainment Follow-up Guides. Of the 
109 interviews completed, only 104 different 
clients were actually seen for follow-uD. 

3. Results of the Study 

For analysis, the one interview conducted 
by the registered nurse has been included in the 
"inexperienced student interviewer" category, due 



to the fact that she was not an experienced fol- 
low-up interviewer with the Program Evaluation 
Project. 

Computation of the two-tailed T-test indicated 
that the difference in mean Goal Attainment scores 
between professional and nonprofessional interview- 
ers and the differences in mean Goal Attainment 
scores for in-person and telephone interviews did 
not reach the p<. 05 level of significance. Table 
XII reports the score analysis of the 109 I'ol low- 
up interviews completed, and Figure IV reports the 
distribution of the 109 Goal Attainment scores. 

0. Follow-up of Clients Te gnnate 

Treatment from the MentlQeaTth Servicl 



1. Introduction to the Study 

This study originated for the purpose of de- 
termining how clients who received no therapy 
after intake or who were recommended for treatment 
but never returned to participate in the assigned 
therapy at the Mental Health Service, scored on 
their Goal Attainment Follow-up Guides. 



2. Implementation of the Study 

Thirty-five clients were followed up for the 
study. Of these, 17 were terminated with no rec- 
ormiendation for further treatment and 18 were as- 
signed to treatment but did not return after their 
initial intake interviews. These two categories 
represent 4.9% of the 708 clients followed up, and 
except for the purposes of this substudy, would 
not have ordinarily been followed up. 

As a result of the two kinds of termination 
assignments, it is assumed that two different 
types of Goal Attainment Follow-up Guides were con- 
structed by the Intake interviewers. For the first 
type, goals were established as appropriate assum- 
ing the client would receive no further treatment 
at either the Mental Health Service or elsewhere, 
and for the second group, goals were established 
with the assumption that the client would receive 
treatment at the Mental Health Service before the 
prescribed follow-up date. 

Table XIII reports the mean comparisons for the 
two groups individually, and the overall figures 
for the entire sample of 35. 

TABLE XIII: Goal Attainment Scores for Two Groups 
of Clients Not Receiving Treatment 
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RECOMMENDED FOR 
TREATMENT: NO 
SHOW 


TERMINATED AFTER 
INTAKE: NO REC- 
OWENOATION FOR 
FURTHER TREATMENT 


THE TWO 

GROUPS 

COMBINED 


N = 18 
X = 56.33 
S.D. 12.64 

S.E. = 2.98 


N = 17 
X = 47.89 
S.D. = 9-63 

S.E. « 2.34 


N =» 35 
X =» 52.23 
S.D. • 11.87 

S.E. - 2.00 


T-test computed for significant difference of means 
yielded difference at the p <.05 level. 
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TABLE XIi: Mean Goal Attainment Scores for In Person and Phone Interviews, 
Experienced and Inexperienced Interviewers 





EXPERIENCED 
INTERVIEWERS 


INEXPERIENCED 
INTERVIEWERS 


OVERALL 


IN 

PFR^DN 


N = 61 
X = 53.22 
c n = in 07 


N = 24 
X = 46.49 
S.D. = 13.fi7 


N = 85 
X = 51.32 
S.D. = 12.10 




S.E. = 1.41 


S.E. = 2.79 


S.E. = 1 .31 


PHONE 


N = 16 
X = 48.61 

c n -- 1 R on 
J. u . - 1 D .yu 


N = 8 

X = 57.69 

S.D. = 13.56 


N = 24 
X = 51.63 
S.D. = 15.49 




S.E. = 3.98 


S.E. = 4.79 


S.E. = 3.16 


TOTAL 


N = 77 
x = 52.26 
S.D. = 12.18 


N = 32 
X = 49.29 
S.D. = 14.30 


N = 109 
X = 51 .39 
S.D. = 12.84 




S.E. = 1.39 


S.E. = 'd.53 


S.E. = \.Z3 



T-test computed for significant differences yielded no significant difference of 
mean scores for the two groups (in person and phone) or for the two types of inter- 
viewers (professional and nonprofessional). 




Goal Attainment FoUow-Up Scores 
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3. Sumnary 



References. 



Clients were eligible for inclusion in the 
study if, at the time of their assignment ("to 
termination"), no referral was made for further 
treatment at another agency. However, 10 clients, 
or 28.57%, were seen by another agency during the 
follow-up period. All but one of the 10 were seen 
by a counseling agency. The mean Goal Attainment 
score for this group is 44.96. The rernaining 25 
clients (71.43%) were not seen by any other 
agency prior to the follow-up interview. The 
mean Goal Attainment score for the group is 55.14. 

Most of the 35 clients followed up were in- 
terviewed by M.S.W. follow-up interviewers. Fif- 
teen clients, or 4%, were followed up by inter- 
viewers of different professional backgrounds 
including B.A. social workers, registered nurses, 
and undergraduate university students. No sig- 
nificant scoring differences were apparent among 
the various types of interviewers. 



Audette, D. & Garwick. Interviewer procedures 
for scoring the goal attainment follow-up 
guide. Unpublished Project Report, July, 
1973. 

Besnett, C. A follow-up interviewer's impression. 
Unpublished Project Report, 1973. 



II Conclusion 

A follow-up program is an integral part of 
any goal -oriented evaluation, the success cf which 
is greatly dependent upon the quality of the data 
and the manner in which it is collected at the 
time of the follow-up interview. 

The experience with Goal Attainment Scaling 
so far suggests that the Goal Attainment Follow- 
up Guide is a viable instrument for follow-up 
interviewing. The form is indeed useful as a 
"follow-up guide". The follow-up process can be 
moderately accurate with inter- rater agreement 
ranging roughly from .50 to .80 depending on 
the client group, the interview medium, and the 
type of interview. 

The follow-up process is flexible and has been 
successfully utilized by telephone or in-person 
interviews, by interviewers from a range of back- 
ground (social work, undergraduates, nursing) and 
by persons with a variety of involvements with 
treatment (research staff, therapist, clients, 
follow-up interviewers). 

Finally* the follow-up process can be used 
to generate information on achievement of goals 
(the Goal Attainment score) on client response 
(consumer satisfaction) and on the relevancy of 
the Goal Attainment Follow-up Guide itself (Goal 
Attainment Follow-up Guide Critique Form). 
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APPENDIX 



I. Interview Schedules. 

Version A - Experimental Follow-up Interview Schedule. 

Version B - Follow-up Interview Schedule Utilized by the Program Evaluation Project Staff during 
July, 1970 - November, 1970. 

Version C - Follow-up Interview chedule Utilized by the Program Evaluation Project Staff during 
November, 1970 - August, 1971. 

Version D - Follow-up Interview Schedule Utilized by the Program Evaluation Project Staff during 
August, 1971 - January, 1973. 

II. Follow-up Forms. 

Form A - Program Evaluation Project Assignment/Appointment Worksheet 

Form B - Program Evaluation Project Patient Consent Form. 

Form C - Follow-up Appointment Worksheet. 

Form D - Program Evaluation Project Collateral Consent Form. 

Form E - Documentation of Elimination from Program Evaluation Project Follow-up. 

Form F - Therapy Session Reporting Form. 
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I. iHTERVltW SCHEOOtES. 

Version A. Enperl rental FoUow-up Interview Schedule . 



PATIENT NAME AND NUlttER . 

ADDRESS . 

TELEPHONE 



FOLLOW-UP INTERVIEWER . 



DATE OF INTAKE _ 
APPOINTMENT TIKE 



1. Statement of purpose in being there. (For example: We are with the 
Hennepin County General Hospital Cental Health Center and we are doinq a 
follow-up on a certain nimbcr of the patients in regard to the service 
they received at the Mental Health Center.) 

2. I understand that you were seen at the Kental Health Center in 

of ? (A yes or no answer will be given. If a no is n-.ven, 

the <:ate will have to be clarified with the patient. 

3. Were you ever seen at the Hennepin County Mental Health Center previous to 
of ? 

If answer is no — Have any members of your fanily been seen there? 
Have your friends been seen there? 

4. Have you been seen by any other agency since °f 

If yes, which one? and are you still being $een there? 

5. Did you have any problems in getting service at the Hennepin County Mental 
Health Center? 

If there is a yes response, but patient is unable to come up wiin any 
concrete reasons, sow suggestions night be: Worker not available, time 
delay, lack of waiting rooms, noisy surroundings, 

6. Were there any services you felt you should receive and didn't? 

7. Were >ou satisfied with the services you received? ' " 
Scale this item according to: 

«) very satisfied ^b) satisfied _ c) Indifferent 

d) dissatisfied e) very dissatisfied 

8. What were your rc<:'.ons for being: 

Satisfied ^ or . 



Dissatisfied 
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Who suggested that you go the the Hennepin County Mental Health Center 
for service? 

How did you feel about going to the Hennepin County Mental Health Center? 
Scale this item according to: 

ai very positively b) positively c) indifferently 

d) negatively e") very negatively 

Mould you return If you felt a need for further services? 

What were the problem which led you to seek service at the Hennepin 
County Mental Health Center? 
Scale Headings are: 



Do you feel any differently about your problems now after having been 
seen at the Hennepin Co^niy Mental Health Center? 
Scale this item according to: 

a) much better b) better c) same d) worse 

e) much worse 
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Version B. Follow- up Interview Schedule Utili zed t>y the Program Evaluation 
Project Staf f' during Ju ly 1970 - Nova:it.er 1970: 



tiATE OF INTAKE 

PATIENT NAME AND NUMBER , 
ADDRESS 



1. (continued) 

the date of intake to 

be sure that both 

interviewer and patient 

are discussing the same 

intake. 



TELEPHONE . 

FOLLOW-UP iNTEkVlEWER 
APPOINTMENT TIME 

PLACE OF Interview 



HENNEPIN COUNTY MENTAL HEALTH CENTER FOLLOW-UP INTERVIEW SCHEDULE 



FOLLOW-UP IffTERVlEWER 
GUIDELINES 


FOLLOW-UP INTERVIEW f.CH[[.iii r 


PATIENT RESPONSE 


Statement of purpose 


I am fron the Hennepin 
County Kental Health Center at 
General Horoital. I believe we 
K)Au» An aPDOlntnrnt for 
today. We at the Mental Health 
Center are attempting to eval- 
uate the services of our program. 
I would like to ask you sonc 
questions about your experience 
with the Center program. 




Optional rerurk if 
the patient seems not 
to understand. 


(You may recall being asked to 
agree to this interview by slgnlnq 
our consent form when you registered 
at the clinic.) 




PART I: Construe r 
satis faction witn' 
servi CCS. 






1. If patient response 
to this question Is 
•no" It will te 
necessary to clarify 


1. I understand that you were seen 
at the Mental Health Centra last 

(date Of intake) 
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1. Yes 

NO 

If w. 

When? 



?. Check patient re- 
sponse if listed. 



*If Sacldl agency, 
indicate name. 



♦If "other", 

npcctfy. 



please 



3. If patient has had 
previous contact with 
the Ctficr, remind him 
that wc ore only Inter- 
ested in his reaction to 
his experiences with the 
Center af ter the Intake 
date speclTTed on the 
schedule. 



Interviewer should 
attempt to sumrurize 
patient's response to 
this Question and arrive 
at an agreed-upon degree 
of willingness to seek 
treatment, which can be 
seal ed. Interviewer 
should record any 
reasons the pdtient 
might give for his 
response. 



Who su^gcstrrl that you ao to the . 
Hennep'.n County Mental Health 

Cent",-? 



Was your decision to seek trej^- 
tncnt a vc'wi/.tary or.c, ar was it 
required (by law. for Instance) 
that you do so? 



Old you have any contact with the 
Mental Health Center previous to 

^ ? 

{Tate of intake) " 



How would you describe your 
feelings about set-king service at 
the Mental Health Center? 
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2. Self 

Doctor 

Fami ly 

member 
Suicide 

phone ^ 

Emergency 

room 
Social 

agency 



Court 
order 

Condition of 
probation, 
parole 

Other 



2a. Voluntary 
Required _ 



3. Yes 



If "yes" 
When? 



4. Very negative, 
very Ambivalent 
about SCeVinq 
treatment 

Negative, some- 
what ambivalent 



Indifferent 

Positive, some- 



4. (continued) 



5. If dniiMer to this 
question is "yes", in- 
ter/ ie-<er inay wish to 
include additional 
comncnt in space be- 
neath question IS. 



6. If there is a "yes" 
response to this ques- 
tion, but patient is not 
able to specify, inter- 
viewer may suggest Itefns 
such ac availability of 
worker, tiaie it took to 
get jppointnent. 
physical Surroundings 
of wJitiRQ dreai etc. 



8. As in question 4, 
intervic.<cr should 
attempt CO sixnndrize 
pdtient riisponse to 
this question dnd 
arrive dt dn dgreeH 
upon dcqrne of 
sdCi sfjction which 
can be scdled in 6a. 



Have you been sc- by dny other 

dgency since ' 

(date oflntjkc) 



Did you hdve dny problens (jetting 
service Jt the Hennepin County 
Mcntdl Health Center? 



Were there dny services you felt 
you should hjve received Jnd didn't? 



Hero you satisfied with the services 
you received? 



wh3t wl 11 Ing to 
seek trcatnent 



Very positive, 
very Milling 



5. Yes 

If "yes"* whcrel 



6. Yes 

If "yes*, list! 



7. Yes 

If "yes", list: 



8a. Very dlssdtis- 
ficd 

Olssdtlsfled 

Indifferent _^ 

Satisfied 

Very Satisfied 



8b. Use ^. ■'! for 
Indlcdtlntj <rJOCl f Ic 
Itons '^ith which the 
patient was Sdtlsflcd 
or dissdtisfled. 



9. If not clearly d 
"yes" or "no" response, 
record any stiitenents 
or qudlifying rcmdrks 
the pdtient might 
provide. 



PART II: Discussion of 
the problems pdtient 
feels prcclpltiitcd his 
going to the Mentdl 
Hedlth Centur. This dls- 
ussion should lead to 
'.he determination of a 
score (-Z. -1 . 0, ♦! , +2) 
for edch of the scales in- 
dicated on the Follow-up 
Guide f llled out for the 
pdtient at Intdke. 

The interviewer should use 
this portion of the inter- 
view lo draw the pjlient 
out in discussing problems, 
rather th in approdchlng the 
probletn o ctly. It is 
possible t)Mt the intdke 
worker defined problem drcds 
dnd goals which the p.itient 
would have defined differ- 
ently. Refrain from 
divulging the mdterldl on 
the grid to the pdtient. 



11. Keeping a list of the 
problems mentioned by the 
pdtient wi 1 1 cndb1 e the 
follow-up Interviewer 
to 'JO back dnd discuss 
whether the pdtient feels 
differently dbout edCh 
problm now. 
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Qb. Whdt were your reasons for being 
dissdtisfled: 



Woulu you return to the Hentdl 
Hcdlth Center If yoj felt a need 
for further service? 



10. What were the problens which led you 
to seek services dt the Hennepin 
County Hentnl Hcdlth Center? 

(SCALE KEAOINGS for this pdtient 
drc: ) 



2. 
3. 
4. . 
5. 

6. . 

7. . 
8- . 
9. . 

10. 



11, Do you feel differently dbout your 
problans now? 



Do you attribute this change in 
feeling to the treatment you 
received at the Center? 



11. Much better^ 

Better 

Sdme 

Worse 

Kjch worse _ 



lid. Yes . 
No 
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If not, to what do you dttrlbutc 
the chdngc? 



4. Pdtitnt's apparent dttltude toward follow-up. 



PART III. Su nr.a ry o f I nte ry i ew . This section should be completed by the interviewer 
dTter tFe 'interview has taken place. Please select the response item 
which you feel best approxina'es your evaludtion of what took place. 
Feel free to qualify yo'Jr ch,-.^ of resp?nr,p by writing in the left hand 
margin if you think t;ual if icdtion is necessary. 



highly fdvordble 
fdvordble 
indifferent 
unfdvordble 
highly unfdvordble 



SUMMARY ITCH 



INTERVIEWER RESPONSE 



1. Length of interview (approximate time) 

less than one half hour 
one hal f to one hour 
one to onf and one hdlf hours 
one dnd one half to two hours 
more thdn two hours 



Qudllty of patient Interdctlon/cooperdtlon In 
interview. 

very responsivti/cooperdtlve 
res pons ive/co'jperdtive 
indifferent 

unrespons ive/uncooperdtive 
very unresponsive/uncooperdtive 



Pidce of interview 

patient' s hor.e 
mcntdl hCdlth center 
P.E.P. office 

patient's pldce of employment 
interviewer's place of employment (if other 

than mental health cuntcr or P.E.P. office) 
public pldCC (other thdn patient's or 

Interviewer's place of employment or 

mental health center) 

specify: 

other 

specify: 



Persons other than interviewer and patient present 
(dudible or visible to Interviewer) dnd partici- 
pating dt interview. Indicate in the space to the 
right the njnber of persons of the specified rela- 
tionship. In the space to the left IndiCdte whet .er 
the other person(s) were merely present or partiri- 
pdting. 

merely ^ present p\rticipat1nq reldtionship 



p.ircnt 

spouse 

child 

roorv-iate 

friend 

(oth*?rT" 



Indlcdte your level of confidence in the scoring 
you did for erUh scdle by rdting 0 for minimun 
confidence, 1 for mo(ierflte confidence, dnd 2 for 
high confidence. 

Scdle 1 
Scdle 2 

Scdle 3 „ 

Scale < 

Scdle 5 _ 



Did *h? patient or your interaction with the 
paliirt L'vitlcr-,rft the possiblity of any new 
n>enti3l/phyr,iLal hejlth problf-v arising since 
intake? Llaborate. 

Yes 
No 



Oid you rule ony referrdls in the course of 
the interview? Yes No 



Whom? _ 
Where? 



If there dre ^ny other significant items which you wish 
to report, please use the space bdlow to describe tiicse: 
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Version C. r»n»^^p lntprvii>w Schedule Utilised by thr Pt^oqrOT Evaluation 
ProlecrStrfrTfurino November t970 r AuMsOOTj. 



DATE OF INTAKE 

PAflENT NAME AND NW«ER . 
ACORESS 



TELEPHONE 

FOLLW-UP INTERVIEWER , 

APPOINTMENT TIME 

PLACE OF INTERVIEW 



L?ir:EPiN courar mental health center follo\i-up i.-iTcv/itw schedule 



FOLLCi-UP INTERVIEWER 
GUIDCLINES 



FOLLOW-uP INTERVIEW SOIEWJLE 



Statement of purpose: 



Optional ronark H 
the patient seems not 
to understand. 



Interviewer should 
state thjt he Is not 
enployed by the Men- 
tal Heilth Center, and 
has no vested Inter- 
ests In the outcome 
of the study. He 
should -ilso infonn 
the patient that the 
results of this Interview 
«!ll not be nade known 
to the patient's therapist 
Interviewer should keep 



I ^ representing 

the Hennepin County Mental Health 
Center at General Hospital. I 
believe that we have an appoinlment 

for today. The Mental Health 

Center is attonptln; to evaluate the 
services of its Program, I would like 
to ask you some question* about your 
experiences with the Center pi-o'jrar>. 

(You may recall being aikc«< to agree 
to this interviL"w by signing our 
consent fom when you registered at 
ihe cl Inlc. ) 



PATIENT RESPONSE 
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6. If there is a "ycr." 
response to this qj'.*..- 
tlon, but patient i'. not 
able to specify, in:cr- 
v1c»<er flwy sugitst iiems 
such availability cf 
worker, time It took to 
9et an appointii^ent, 
physical surroundings 
of wai ting area, etc. 


6. Did yDu h.^vc any rroblc<:i«i golting 
service <ii the Hpnncinn County 
Mental Hoalth Center? 


6. Yfs 

If "yes", list: 

No 


7. Sa,-ne as Form 9. 


7. Same as Form B. 


7. Same as Form B. 


8. As in question 4, 
Interviewer should 
atttnpt to surriarize 
patient response lo 
this question and 
arrive at an «grtcd 
upon dejree of 
satisfaction whicli 
can be scaled. 


S. How Satisfied were you w. the 
services you received? 


6. Very dissatisfied 

Dissatisfied 

Indi fferent 

Satisfied 

Very Satisfied 


Ba. U'^e space fcr 
indicating soccif'C 
iteiTS with wJ.ich the 
patient was dJSSdti^- 
fUd or satisfied. 


Sa. UhAi vicre your reasons for being 
dissdtisf iedr 

satisfied: 






9. Are you still reccivini trc.U- 
ment at the Center? 


9. Yes 

No 


9a. If ro*. clcjT 1> 
■yei" O"" ""O ' rtsf ' '!^'-, 
record any stateci'.riis 
or qualifying rtrjrt.s 
the patient might 
provide. 


9a. If no, wo'jld you rtt.urn to the 
Mental Health Center if you 
fcU a need for further 
Service? 


9.1. Yes 

Ho 



In mind th«t it Is 
preferred that he not be 
informed of the patient's 
therapist or therapy, and 
(If it seems appropriate) 
he might ask the patient 
not to divulge this 
infonnation. 






PART I : Consumer 
satisfaction \jith 
services. 

1 . Same as Form B. 


1. Same as Form B. 


1. Same as Fom B. 


2. Same as Fom B. 


2. Same as Fom B. 


2. Same as Fom B. 




2a. Same as Fom B. 


2a. Same as Fom B. 


3. If patient has had 
previous contact with 
the Center, rcjnind him 
that we arc only int- 
crcslcd ir. his reaction 
to his experience with 
the Center after the 
intake dutc specified 
on the schedule. 
(Please note that we 
consider A-3, General's 
inpatient unit* to be 
an agency separate 
from the Mental 
Health Center.) 


3. Did you have any contact with 
the Mental Health Center prev- 
ious to 7 
(date of intakoj 


3. Yes 

If "yes". 
When? 

Ho 


4. Scime as i-i>,><« b. 


4. Same as fonn B 


4. Sfljiie as Fom 6. 


5. If answer to this 
question is "yes", in- 
terviewer may wish to 
include additional 
conwcnt in space be- 
neath question rS. 

Again note that A-3 
is defined as a sep- 
arate agency (see #3). 


5. Have you been seen by any other 
agency since ^ ? 

(date of intdkcj 


5. Yes 

If "yes", where? 

No 
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PART II : Discussion of 
the problems the patient 
feels precipitated his 
going to the Mental 
Health Center. This dis- 
cussion should lead to 
the determination of a 
score (-2, -1. 0, +1, +2) 
for each of the staler in- 
dicated on the follow-up 
guide filled out for the 
patient at intake. These 
scores should be indic- 
ated on the Gui de by a 
check or asterisk. 

The interviewer should use 
this portion of the interview 
to draw the patient out in 
discussing probletas, 
rather than approaching the 
problem directly. It is 
r issible I'^jt the intal^e 
wo'-ker ti' - ined problem 
are.i'i goals which the 
patient would have defined 
differently. Refrain from 
divulging the material on 
the Guide to the patient 
except in a very genera' 
sense if this seoiis 
necessary. 

In the space to the right, 
specify those problem 
areas the patient indicated 
as being problematic and 
bringing him to the Center 
for tredtnent. 
Keeping a list of 
probltns mentioned by 
the patient wi 1 1 enable 
the follow-up interviewer 
to go back and discuss 
whether the patient 
feels differently about 
each pi'oblan now. 
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What were the problems which led you 
to seek services at the Hennepin County 
Mental Health Center? 

(SCALE HEADINGS for this patient are:) 



9. 
10. 



1. 
2. 
3. 
4. 
5. 
6. 
7. 
8. 
9. 
10. 



U. Do you feel differently about your 
problems now? 
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n . Much better , 

Better 

S?jiie 

Worse 

Much worse 
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11a. Interviewer should 
•ttcnpt to Stfividrlze 
the patient's response 
to this question. 



Ua. Do you attribute this chanqc or 
lack of change in feeling to th- 
treatment you received at the 
Center? 



lib. If not mostly, to what (or what 
else) do you attribute it? 



lla. Yes» mostly 

Yes , partly 

Not for the most 
part 

Not at all 



PART III. SU?»tARY or INTERVIEW . 



PLEASE NOTE : Suinnary of Interview quesLions 1 
through 8, are the sa-ne as in 
Foro B. 



In spite of your effort!, to remain uninformed 
of the therapy and therjpist(s) involved, did 
this infomation become known to you? 

Yes No 

If yes, please specify: 



preferred that he not 
be infomi&d of tht* 
patient's therapy or 
therapist, and (if it 
seeits appropriate) he 
might ask the patient 
not to divulge this 
information. 



PART I: Consu mer PLEASE NOTE: Questions 1,2, 4. and 7 are the sane as in Form ! 

satififjction wuh Questions 2a, 3, 5, 6, 8, 8a, 9 are the same as in 

scrv i'c cs . Form C. 



PART I I : Di scussion of 
patient prpblw is. 



PLEAS L NOTL: Questions 10,11,11a, and lib are the same 
as in Form C. 



Do you have any suggestions for 
improving the operation of the 
Hennepin County Hental Hfolth 
Center, in addition to the 
cements you have already made? 



PART 111- Sutnn a ry of Intrrview . 

PLEASE NOTE: 



Summary of interview questions 1 
through 9, arc the same as in 
Form C. 



Patient Name: 



Follow-up Interviewer: 



Negotiated 

Guide Number 



Version p. Follow-uP Int ervi ew Schedul e Utilized by the Program Evaluation 
Project Staff during August 1971 - January 1973 . 



DATE OF INTAKE _ 
NAME AND NUMBER . 

ADDRESS 

TELEPHONE 



AT TIME OF IHTAKE: 




AGE SEX 


MARITAL STATUS 




FOLLOU-UP INTERVIEWER 






APPOINTMENT TIME 






Piacr OF INTERVIEW 




HENNEPIN COUNTY MENTAL HEALTH CEKTER FOLLOW-UP INTERVIEW SCHEDULE 


FOLLOW-UP INTERVIEWER 


FOLLOW-UP INTERVIEW SCHEDULE 


PATIEKT RESPONSE 


Statement of purpose: 


I gm representing 
the Hennepin County Hental Health 
Center at General Hospital. I 
believe we have an ki>^ointment 
for today. The 
Mental Health Center is attempting 
to evaluate the services of its 
Program. I would like to ask you 
some questions about your 
experience with the Center program. 




Optional renark If 
the patient seems not 
to understand. 

Interviewer should 
state that he is not 
employed by the 
Mental Health Center, 
and has no vested 
interests in the 
outcone of the study. 


(You may recall being asked to 
agree to this interview by signing 
our consent form when you 
registered at the clinic.) 




Interviewer should keep 
in mind that it is 
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PATIENT NAMt AND NUMDER 

THERAPIST(S) INVOLVED IN TREATMENT OF PATIENT (to be filled in by research staff 
after completion of the follow-up intervitv/) 



FOLLOW-UP INTERVIEWER 

TELEPHONE NUMBER WHERE If(TERVIEWER CAN BE COKTAHED FOR ADDITIONAL IHFORMATIDH. IF 
NECESSARY ♦ 

(NOTE TO THE FOLLOW-UP INTERVIEWER: a copy of this sheet will be forwarded to the patient's 
therapist If action or intervention by the Mental Health Service is recommended.) 

8ASED ON YOUR INTERACTION WITH THE PATIENT, DO YOU RECOfflEND ACTION BY THE MENTAL 
HEALTH SERVICE AT THIS TIME? YES NO 

PLEASE EXPLAIN FULLY: 



WAS THE PATIENT INFORMED THAT YOU '.JOULO LIKE TO NOT I Pf THE HEKTAL HEALTH SERVICE 01 
THE SITUATION DESCRIBED ABOVE, A.SD THAT A CENTER STAFF MEMBFR MIDtT CO»<TACT HIM? 

YES NO 

IF YES. WHAT WAS THE PATIF'fT'S RFSPON*;' 



Scale 1 Sc ale 2 Scale 3 Scale ^ Scale 5 

1) Are the scales „ ^ w u 
relevant? Yes No Yes No Ye; No Yes No Yes No 

2) Are the scale Optimistic Optimistic Optimistic Optimistic Optimistic 
levels realistic? Right On Right On Ri'Jht On Right On Right On 
(Circle one) Pessimistic Pessimistic Pessimistic Pessimistic Pessimistic 

3) would you have added any scales to this guide? Yes No 

4) If you answered "yes" to question 3, please indicate what additional scale headings 
or content areas of existin-j scales you would have added. 



PLEASE NOTE : At this point in the interview 
schedule, a blank sheet of 
paper is included as a suimary 
sheet for additional connents. 
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II. FOLLOW-UP FORMS. 



TCIIPHONC XlMW ^ 

ACiOtllONAL COITACT lNf3(>1AT]0N: 
(t.9. rtftrrtl town*, other •9«*<y 
InvolviMnt) 



JUSTIMCATICM: 



_ C««l»nt fOn» Kal Signed 

T-»0 Afur 1 Inirnrirwi 
~ M ur 2 » C'td 



r*i{snt tc AO 

UKAAl^: 



(Including (ny diiPvti. 
ip«cl«l ducjfiton, 

p«tt«nt'i 4fitgr«tcnt) 



FIRST iKTAtt iHTtmin OH: _J. L_ 

HO. Of TCttOM 



FATirKT ASSIGMQ: 



_ NoA'rindeity 



♦r. CAIL TO DOlPfiJKt TH£Wl$T 



PATILHT'S AM: SI 

iiaul WOWIR: 

IKTAtt MRUI'S KtamnOATtON: 



A.l. TO MUf CONTACT 



A.I. TO CHJCK WITK 

■^0 ccTiwtNi If cwmcrrwH 



COtaACT COW'LniD iT: 



_ t>*Tt THIIUPIST COrnACTtO / / 



TMIWISI AVAILWl: 



w/A'.ir7m t?T75XTr/nw wy/wirAiKi 

HOW WAS TMt Mllfl^ fr.'''n.'07 ^ ^ . . 

7cT^^V^7~€^^^'lc^'.'Jr:.*"rTJ^^cr given? tht •OJreiiT Ple»ir k**P * cof»tlnutf)g rrcort.) 



MA'.t v.-:«.i«iJf Toua i*Jifn.M:TiON witi thi p>':L1JNI- , . . 

rrVq:'t«:t<"flric*^<'r«i;ir"^ta to cirSiC for ther«r/ iiiignetf. jwtl»nt rriUtrt 

in*ol*trtr.t In tic t*>erJ!.y jiitgntd «nd lonr diicuiiion iniued. iitlMte Mhfther or 
nnt tft« pjtltot kUI fulltTrf through on lingwnt. /ny Jpectil problem thi p*tlent 
wcntfoMd) 



PaHIKT ACCtniO AWOtfnKfNT Ofl: 
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THERAPIST lOTIftIO „ 



Follow-up Forn C. Follow-up AppolntJncnt Worksheet . 



INTAKE WORKER: 

DATE OF INTAKE: 
I of INTAKE: 
NAME: 

PARENTS OR 
HUSBAND'S nM^^: 

AOOR£SS; 

telephone: 

DATE OF INTERVIEW: 

PLACE OF INTERVIEW: 

INTERVIEWER: 



HO. OF ACCEPT.: 



PATTERN 

ASSIGJiMENfrR - NR RELIABILITY STUDY:, 

ASSICiHENT DATE: 

CONSENT SI WED: 

CL'RP£ffTLY l?f THEP/J^Y: 

NUMBER OF VISITS: 

THERAPIST: TREATMEKT: 



FOLLOW-UP DATE: 



APPOINTMENT CONFIRMED: 

WITH PATIENT INTERVIEWER 

SEND REMINDER 

MATERIALS SENT TO INTERVIEWER 

INTERVIEW COMPLHED: 

MATERIALS RETURNED: 

MATERIALS COHPLHE: 



DATE OF BIRTH . 
SOC. SEC. NO. 



_ MAR. STATUS ^ 
SEX 



Follow-up Fom B. Prograa Evaluation Project PatUnt Consent Form 



HENNEPIN COUNTY GENERAL HOSPITAL 
PROGRAH EVALUATION PROJECT 
PATIEfCT CONSENT 



Date 



I Agree to participate in the Progrjin Evaluation Research 
Project which is now in progress at Hennepin County Hental 
Health Center. I understand that this is a scientifically 
structured study undertaken to determine the effectiveness of 
the Mental Hejlth Center program. Because of this, I agree to 
pemit the Kental Health Center to share relevant records with 
the Program tveluftion Project Staff and to let a social 
worker coniact -e lattr on to find out if I have benefited 
fron rr^ therapy. 



Signed 



Address 
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Follow-up Form 0. Profiram Evaluation Project Collateral Consent Fonn 



HENNEPIN COUNTY GENERAL HOSPITAL 
PROGRAM EVALUATION PROJECT 
CONSENT FORM 



I give my permission to the Program Evaluation Research Project 
to contact the individuals and/or organizations I have indicated 
below. I understand that the Project will be doing this for the 
purpose of assessing the effectiveness of the treatment I received 
at the Mental Health Center. 



SIMWRIZE PATIENT CONTACT: 

(include when, where, ar.d patient's response) 



IfCTERVIEW NOT COMPLETED. K< 
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Persons or organizations that can be contacted: 

1. 

2. 

3. 

4. 

5. 
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Follow-up Form E. Documentation of Elimination from Program Evaluation Project Follow-up . 



TPTtTerTt'nrme) (inoT ofaccepTT ImodeT" 

(hospTtaTlibT) 7cTc"priot"stu3y no.) TtFerapistT 

Tmo'.~eTigTFle*Tor follow-up) TeTinmrarTon code) Tpattern) 



Circle the code number of the appropriate reason for follow-up elimination: 



GOAL ATTAINMENT FOLLOW-UP GUIDE 

100— No G.A.F.G. completed 

101 — G.A.F.G. completed after three intakes 

102— G.A.F.G. completed after patient 
entered treatment 

103- -Prestudy patient 

104— Patient previously included in PEP 
study and followed-up 

105- -Invalid follow-up guide 
COURSE OF THERAPY 

200— Zero therapy sessions completed in 
the assigned mode 

201— One therapy session completed and 
patient terminated from treatment 

202— One therapy session completed and 
patient transferred to another mode 

203— One therapy session completed and 
therapist does not anticipate 
another session 

204— Other 
PATIENT CONSE NT 

300- -Patient refu-^es to sign PEP consent 

301- Patient has signed consent, but re- 
fuses interview at time of follow-up 

302- Other 

PATIENT CANNOT BE CONTACTED 

4jO--Patient deceased at time of follow-up 

401 — -Patient has moved out of state 

402— Patient has moved out of Metro area 
and is not reachable 

403— PEP is unable to locate patient 



LOW PRIORITY FOR FOLLOW-UP 

500— Random assignment to Individual 

501— -Random assignment to Group 

502— Random assignment to Day Center 

503— Random assignment to Drug Clinic 

504— -Random assignment to Drug Therapy 

505— -Nonrandom assignment to Individual 

506— -Nonrandom assignment to Group 

507— -Nonrandom assignment to Day Center 

508— Nonrandom assignment to Drug Clinic 

509— Nonrandom assignment to Drug Therapy 

510— Nonrandom assignment to Marriage Counsel. 

511 — -Nonrandom assignment to PRN 

512— Nonrandom assignment to MPC/APC 

513— Nonrandom assignment to Child Unit. 

514— T-90 to Annex 3 (In-patient Service) 

515— T-90, 1 interview to another agency 

516— T-90, 2 interviews to another agency 

517— T-90, 1 interview, (no reference) 

518— -T-90, 2 interviews, (no reference) 

519— T-90, 1 interview, evaluation only 

520— -T-90, 2 interviews, evaluation only 



OTHER 

600— Random patient eligible for follow-up, 

but eliminated on the basis of cost 
700-Other 

CRISIS INTERVENTION CErfTER 

800— Unable to locate patient 

801— Patient refuses follow-up 

802— CIC stipulates no follow-up required 

803— -NO G.A.F.G. completed 

804— Post CIC treatment at another agency 
makes follow-up unfeasible 



DRUG STUDY 

900- Zero therapy sessions completed 

901- -0n therapy session completed and patient 

dropped out. 

902- Two therapy sessions completed and patient 
dropped out. 

903- -Three therapy sessions completed and patient 

dropped out 

904- Patient transferred to another therapy mode after 
one or more therapy sessions 

905- -Patient has signed consent form, but reconsiders 

and refuses follow-up 

906- -Patient is deceased at time of follow-up 

907- -Patient has moved out of state 

908- -PEP is unable to locate patient 

909- Deteriorization 

910- Toxicity 



Patient eliminated from follow-up on 

fdite] 

Patient eliminated from follow-up by ^ 

(name) 
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Follow-up Fonii F 



Therapy Session Reporting Form . 



P 
E 
P 



VALUATION 



ROJECT 



O 



MEMO TO: 



FROM: 



Donna Audette, P.E.P. 



DATE: 



Our records show that the follrvn'ng patient: 



(Patient Name) 



(HCGH No J 



has been assigned to you for therapy. We have been unable to locate this 
patient's Mental Health Center chart; or, have not been able to determine 
from the available chart the number of therapy sessions completed. We 
would very much appreciate your completing the following for us. 



Our records show the patient's Assigned Therapy Mode 

to be Is this correct? Yes No 

Is the patient still in therapy? Yes No 

How many tharapy sessions has the patient completed to date? — 
How many additional sessions do you anticipate? — 

Additional Comments: , 



Thank you. 



the mcgiil building — fifth floor -501 paik avenue $.— minnea|x,Iis — minnewta — 55415 — (612) 
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